diff --git a/docs/zh/12-taos-sql/02-database.md b/docs/zh/12-taos-sql/02-database.md index e3a0aa7c87461fd1621a38093871a1542e3dbf98..38119dfb445fb8e2ae920bf2b41d1b1dc8fffd2a 100644 --- a/docs/zh/12-taos-sql/02-database.md +++ b/docs/zh/12-taos-sql/02-database.md @@ -6,53 +6,85 @@ description: "创建、删除数据库,查看、修改数据库参数" ## 创建数据库 -``` -CREATE DATABASE [IF NOT EXISTS] db_name [KEEP keep] [DAYS days] [UPDATE 1]; -``` - -:::info -1. KEEP 是该数据库的数据保留多长天数,缺省是 3650 天(10 年),数据库会自动删除超过时限的数据; -2. UPDATE 标志数据库支持更新相同时间戳数据;(从 2.1.7.0 版本开始此参数支持设为 2,表示允许部分列更新,也即更新数据行时未被设置的列会保留原值。)(从 2.0.8.0 版本开始支持此参数。注意此参数不能通过 `ALTER DATABASE` 指令进行修改。) - 1. UPDATE 设为 0 时,表示不允许更新数据,后发送的相同时间戳的数据会被直接丢弃; - 2. UPDATE 设为 1 时,表示更新全部列数据,即如果更新一个数据行,其中某些列没有提供取值,那么这些列会被设为 NULL; - 3. UPDATE 设为 2 时,表示支持更新部分列数据,即如果更新一个数据行,其中某些列没有提供取值,那么这些列会保持原有数据行中的对应值; - 4. 更多关于 UPDATE 参数的用法,请参考[FAQ](/train-faq/faq)。 -3. 数据库名最大长度为 33; -4. 一条 SQL 语句的最大长度为 65480 个字符; -5. 创建数据库时可用的参数有: - - cache: [详细说明](/reference/config/#cache) - - blocks: [详细说明](/reference/config/#blocks) - - days: [详细说明](/reference/config/#days) - - keep: [详细说明](/reference/config/#keep) - - minRows: [详细说明](/reference/config/#minrows) - - maxRows: [详细说明](/reference/config/#maxrows) - - wal: [详细说明](/reference/config/#wallevel) - - fsync: [详细说明](/reference/config/#fsync) - - update: [详细说明](/reference/config/#update) - - cacheLast: [详细说明](/reference/config/#cachelast) - - replica: [详细说明](/reference/config/#replica) - - quorum: [详细说明](/reference/config/#quorum) - - comp: [详细说明](/reference/config/#comp) - - precision: [详细说明](/reference/config/#precision) -6. 请注意上面列出的所有参数都可以配置在配置文件 `taosd.cfg` 中作为创建数据库时使用的默认配置, `create database` 的参数中明确指定的会覆盖配置文件中的设置。 - -::: +```sql +CREATE DATABASE [IF NOT EXISTS] db_name [database_options] + +database_options: + database_option ... + +database_option: { + BUFFER value + | CACHEMODEL {'none' | 'last_row' | 'last_value' | 'both'} + | CACHESIZE value + | COMP {0 | 1 | 2} + | DURATION value + | FSYNC value + | MAXROWS value + | MINROWS value + | KEEP value + | PAGES value + | PAGESIZE value + | PRECISION {'ms' | 'us' | 'ns'} + | REPLICA value + | RETENTIONS ingestion_duration:keep_duration ... + | STRICT {'off' | 'on'} + | WAL {1 | 2} + | VGROUPS value + | SINGLE_STABLE {0 | 1} + | WAL_RETENTION_PERIOD value + | WAL_ROLL_PERIOD value + | WAL_RETENTION_SIZE value + | WAL_SEGMENT_SIZE value +} +``` + +### 参数说明 +- buffer: 一个 VNODE 写入内存池大小,单位为MB,默认为96,最小为3,最大为16384。 +- CACHEMODEL:表示是否在内存中缓存子表的最近数据。默认为none。 + - none:表示不缓存。 + - last_row:表示缓存子表最近一行数据。这将显著改善 LAST_ROW 函数的性能表现。 + - last_value:表示缓存子表每一列的最近的非 NULL 值。这将显著改善无特殊影响(WHERE、ORDER BY、GROUP BY、INTERVAL)下的 LAST 函数的性能表现。 + - both:表示同时打开缓存最近行和列功能。 +- CACHESIZE:表示缓存子表最近数据的内存大小。默认为 1 ,范围是[1, 65536],单位是 MB。 +- COMP:表示数据库文件压缩标志位,缺省值为 2,取值范围为 [0, 2]。 + - 0:表示不压缩。 + - 1:表示一阶段压缩。 + - 2:表示两阶段压缩。 +- DURATION:数据文件存储数据的时间跨度。可以使用加单位的表示形式,如 DURATION 100h、DURATION 10d等,支持 m(分钟)、h(小时)和 d(天)三个单位。不加时间单位时默认单位为天,如 DURATION 50 表示 50 天。 +- FSYNC:当 WAL 参数设置为2时,落盘的周期。默认为3000,单位毫秒。最小为0,表示每次写入立即落盘;最大为180000,即三分钟。 +- MAXROWS:文件块中记录的最大条数,默认为4096条。 +- MINROWS:文件块中记录的最小条数,默认为100条。 +- KEEP:表示数据文件保存的天数,缺省值为 3650,取值范围 [1, 365000],且必须大于或等于 DURATION 参数值。数据库会自动删除保存时间超过KEEP值的数据。KEEP 可以使用加单位的表示形式,如 KEEP 100h、KEEP 10d 等,支持m(分钟)、h(小时)和 d(天)三个单位。也可以不写单位,如 KEEP 50,此时默认单位为天。 +- PAGES:一个 VNODE 中元数据存储引擎的缓存页个数,默认为256,最小64。一个 VNODE 元数据存储占用 PAGESIZE * PAGES,默认情况下为1MB内存。 +- PAGESIZE:一个 VNODE 中元数据存储引擎的页大小,单位为KB,默认为4 KB。范围为1到16384,即1 KB到16 MB。 +- PRECISION:数据库的时间戳精度。ms表示毫秒,us表示微秒,ns表示纳秒,默认ms毫秒。 +- REPLICA:表示数据库副本数,取值为1或3,默认为1。在集群中使用,副本数必须小于或等于 DNODE 的数目。 +- RETENTIONS:表示数据的聚合周期和保存时长,如RETENTIONS 15s:7d,1m:21d,15m:50d表示数据原始采集周期为15秒,原始数据保存7天;按1分钟聚合的数据保存21天;按15分钟聚合的数据保存50天。目前支持且只支持三级存储周期。 +- STRICT:表示数据同步的一致性要求,默认为off。 + - on 表示强一致,即运行标准的 raft 协议,半数提交返回成功。 + - off表示弱一致,本地提交即返回成功。 +- WAL:WAL级别,默认为1。 + - 1:写WAL,但不执行fsync。 + - 2:写WAL,而且执行fsync。 +- VGROUPS:数据库中初始vgroup的数目。 +- SINGLE_STABLE:表示此数据库中是否只可以创建一个超级表,用于超级表列非常多的情况。 + - 0:表示可以创建多张超级表。 + - 1:表示只可以创建一张超级表。 +- WAL_RETENTION_PERIOD:wal文件的额外保留策略,用于数据订阅。wal的保存时长,单位为s。默认为0,即落盘后立即删除。-1表示不删除。 +- WAL_RETENTION_SIZE:wal文件的额外保留策略,用于数据订阅。wal的保存的最大上限,单位为KB。默认为0,即落盘后立即删除。-1表示不删除。 +- WAL_ROLL_PERIOD:wal文件切换时长,单位为s。当wal文件创建并写入后,经过该时间,会自动创建一个新的wal文件。默认为0,即仅在落盘时创建新文件。 +- WAL_SEGMENT_SIZE:wal单个文件大小,单位为KB。当前写入文件大小超过上限后会自动创建一个新的wal文件。默认为0,即仅在落盘时创建新文件。 ### 创建数据库示例 -创建时间精度为纳秒的数据库, 保留 1 年数据: - ```sql -CREATE DATABASE test PRECISION 'ns' KEEP 365; -``` - -## 显示系统当前参数 +create database if not exists db vgroups 10 buffer 10 ``` -SHOW VARIABLES; -``` -## 使用数据库 +以上示例创建了一个有 10 个 vgroup 名为 db 的数据库, 其中每个 vnode 分配也 10MB 的写入缓存 + +### 使用数据库 ``` USE db_name; @@ -63,61 +95,42 @@ USE db_name; ## 删除数据库 ``` -DROP DATABASE [IF EXISTS] db_name; +DROP DATABASE [IF EXISTS] db_name ``` -删除数据库。指定 Database 所包含的全部数据表将被删除,谨慎使用! +删除数据库。指定 Database 所包含的全部数据表将被删除,该数据库的所有 vgroups 也会被全部销毁,请谨慎使用! ## 修改数据库参数 -``` -ALTER DATABASE db_name COMP 2; -``` - -COMP 参数是指修改数据库文件压缩标志位,缺省值为 2,取值范围为 [0, 2]。0 表示不压缩,1 表示一阶段压缩,2 表示两阶段压缩。 - -``` -ALTER DATABASE db_name REPLICA 2; -``` - -REPLICA 参数是指修改数据库副本数,取值范围 [1, 3]。在集群中使用,副本数必须小于或等于 DNODE 的数目。 - -``` -ALTER DATABASE db_name KEEP 365; -``` - -KEEP 参数是指修改数据文件保存的天数,缺省值为 3650,取值范围 [days, 365000],必须大于或等于 days 参数值。 - -``` -ALTER DATABASE db_name QUORUM 2; -``` - -QUORUM 参数是指数据写入成功所需要的确认数,取值范围 [1, 2]。对于异步复制,quorum 设为 1,具有 master 角色的虚拟节点自己确认即可。对于同步复制,quorum 设为 2。原则上,Quorum >= 1 并且 Quorum <= replica(副本数),这个参数在启动一个同步模块实例时需要提供。 - -``` -ALTER DATABASE db_name BLOCKS 100; -``` - -BLOCKS 参数是每个 VNODE (TSDB) 中有多少 cache 大小的内存块,因此一个 VNODE 的用的内存大小粗略为(cache \* blocks)。取值范围 [3, 1000]。 - -``` -ALTER DATABASE db_name CACHELAST 0; -``` - -CACHELAST 参数控制是否在内存中缓存子表的最近数据。缺省值为 0,取值范围 [0, 1, 2, 3]。其中 0 表示不缓存,1 表示缓存子表最近一行数据,2 表示缓存子表每一列的最近的非 NULL 值,3 表示同时打开缓存最近行和列功能。(从 2.0.11.0 版本开始支持参数值 [0, 1],从 2.1.2.0 版本开始支持参数值 [0, 1, 2, 3]。) -说明:缓存最近行,将显著改善 LAST_ROW 函数的性能表现;缓存每列的最近非 NULL 值,将显著改善无特殊影响(WHERE、ORDER BY、GROUP BY、INTERVAL)下的 LAST 函数的性能表现。 +```sql +ALTER DATABASE db_name [alter_database_options] + +alter_database_options: + alter_database_option ... + +alter_database_option: { + CACHEMODEL {'none' | 'last_row' | 'last_value' | 'both'} + | CACHESIZE value + | FSYNC value + | KEEP value + | WAL value +} +``` + +:::note +其它参数在3.0.0.0中暂不支持修改 -:::tip -以上所有参数修改后都可以用 show databases 来确认是否修改成功。另外,从 2.1.3.0 版本开始,修改这些参数后无需重启服务器即可生效。 ::: -## 显示系统所有数据库 +## 查看数据库 + +### 查看系统中的所有数据库 ``` SHOW DATABASES; ``` -## 显示一个数据库的创建语句 +### 显示一个数据库的创建语句 ``` SHOW CREATE DATABASE db_name; @@ -125,3 +138,4 @@ SHOW CREATE DATABASE db_name; 常用于数据库迁移。对一个已经存在的数据库,返回其创建语句;在另一个集群中执行该语句,就能得到一个设置完全相同的 Database。 +### 查看数据库参数 diff --git a/docs/zh/12-taos-sql/03-table.md b/docs/zh/12-taos-sql/03-table.md index d7235f312933ec46ed427d5da7e2c5a229fa2926..9f3586ce6dc594725a01994d95be94ec843a55fc 100644 --- a/docs/zh/12-taos-sql/03-table.md +++ b/docs/zh/12-taos-sql/03-table.md @@ -2,13 +2,45 @@ title: 表管理 --- -## 创建数据表 - -``` -CREATE TABLE [IF NOT EXISTS] tb_name (timestamp_field_name TIMESTAMP, field1_name data_type1 [, field2_name data_type2 ...]); -``` - -:::info 说明 +## 创建表 + +`CREATE TABLE` 语句用于创建普通表和以超级表为模板创建子表。 + +```sql +CREATE TABLE [IF NOT EXISTS] [db_name.]tb_name (create_definition [, create_definitionn] ...) [table_options] + +CREATE TABLE create_subtable_clause + +CREATE TABLE [IF NOT EXISTS] [db_name.]tb_name (create_definition [, create_definitionn] ...) + [TAGS (create_definition [, create_definitionn] ...)] + [table_options] + +create_subtable_clause: { + create_subtable_clause [create_subtable_clause] ... + | [IF NOT EXISTS] [db_name.]tb_name USING [db_name.]stb_name [(tag_name [, tag_name] ...)] TAGS (tag_value [, tag_value] ...) +} + +create_definition: + col_name column_definition + +column_definition: + type_name [comment 'string_value'] + +table_options: + table_option ... + +table_option: { + COMMENT 'string_value' + | WATERMARK duration[,duration] + | MAX_DELAY duration[,duration] + | ROLLUP(func_name [, func_name] ...) + | SMA(col_name [, col_name] ...) + | TTL value +} + +``` + +**使用说明** 1. 表的第一个字段必须是 TIMESTAMP,并且系统自动将其设为主键; 2. 表名最大长度为 192; @@ -18,106 +50,147 @@ CREATE TABLE [IF NOT EXISTS] tb_name (timestamp_field_name TIMESTAMP, field1_nam 6. 为了兼容支持更多形式的表名,TDengine 引入新的转义符 "\`",可以让表名与关键词不冲突,同时不受限于上述表名称合法性约束检查。但是同样具有长度限制要求。使用转义字符以后,不再对转义字符中的内容进行大小写统一。 例如:\`aBc\` 和 \`abc\` 是不同的表名,但是 abc 和 aBc 是相同的表名。 需要注意的是转义字符中的内容必须是可打印字符。 - 上述的操作逻辑和约束要求与 MySQL 数据的操作一致。 - 从 2.3.0.0 版本开始支持这种方式。 -::: +**参数说明** +1. COMMENT:表注释。可用于超级表、子表和普通表。 +2. WATERMARK:指定窗口的关闭时间,默认值为 5 秒,最小单位毫秒,范围为0到15分钟,多个以逗号分隔。只可用于超级表,且只有当数据库使用了RETENTIONS参数时,才可以使用此表参数。 +3. MAX_DELAY:用于控制推送计算结果的最大延迟,默认值为 interval 的值(但不能超过最大值),最小单位毫秒,范围为1毫秒到15分钟,多个以逗号分隔。注:不建议 MAX_DELAY 设置太小,否则会过于频繁的推送结果,影响存储和查询性能,如无特殊需求,取默认值即可。只可用于超级表,且只有当数据库使用了RETENTIONS参数时,才可以使用此表参数。 +4. ROLLUP:Rollup 指定的聚合函数,提供基于多层级的降采样聚合结果。只可用于超级表。只有当数据库使用了RETENTIONS参数时,才可以使用此表参数。作用于超级表除TS列外的其它所有列,但是只能定义一个聚合函数。 聚合函数支持 avg, sum, min, max, last, first。 +5. SMA:Small Materialized Aggregates,提供基于数据块的自定义预计算功能。预计算类型包括MAX、MIN和SUM。可用于超级表/普通表。 +6. TTL:Time to Live,是用户用来指定表的生命周期的参数。如果在持续的TTL时间内,都没有数据写入该表,则TDengine系统会自动删除该表。这个TTL的时间只是一个大概时间,我们系统不保证到了时间一定会将其删除,而只保证存在这样一个机制。TTL单位是天,默认为0,表示不限制。用户需要注意,TTL优先级高于KEEP,即TTL时间满足删除机制时,即使当前数据的存在时间小于KEEP,此表也会被删除。只可用于子表和普通表。 -### 以超级表为模板创建数据表 +## 创建子表 -``` +### 创建子表 + +```sql CREATE TABLE [IF NOT EXISTS] tb_name USING stb_name TAGS (tag_value1, ...); ``` -以指定的超级表为模板,指定 TAGS 的值来创建数据表。 +### 创建子表并指定标签的值 -### 以超级表为模板创建数据表,并指定具体的 TAGS 列 - -``` +```sql CREATE TABLE [IF NOT EXISTS] tb_name USING stb_name (tag_name1, ...) TAGS (tag_value1, ...); ``` -以指定的超级表为模板,指定一部分 TAGS 列的值来创建数据表(没被指定的 TAGS 列会设为空值)。 -说明:从 2.0.17.0 版本开始支持这种方式。在之前的版本中,不允许指定 TAGS 列,而必须显式给出所有 TAGS 列的取值。 +以指定的超级表为模板,也可以指定一部分 TAGS 列的值来创建数据表(没被指定的 TAGS 列会设为空值)。 -### 批量创建数据表 +### 批量创建子表 -``` +```sql CREATE TABLE [IF NOT EXISTS] tb_name1 USING stb_name TAGS (tag_value1, ...) [IF NOT EXISTS] tb_name2 USING stb_name TAGS (tag_value2, ...) ...; ``` -以更快的速度批量创建大量数据表(服务器端 2.0.14 及以上版本)。 +批量建表方式要求数据表必须以超级表为模板。 在不超出 SQL 语句长度限制的前提下,单条语句中的建表数量建议控制在 1000 ~ 3000 之间,将会获得比较理想的建表速度。 -:::info +## 修改普通表 -1.批量建表方式要求数据表必须以超级表为模板。 2.在不超出 SQL 语句长度限制的前提下,单条语句中的建表数量建议控制在 1000 ~ 3000 之间,将会获得比较理想的建表速度。 +```sql +ALTER TABLE [db_name.]tb_name alter_table_clause + +alter_table_clause: { + alter_table_options + | ADD COLUMN col_name column_type + | DROP COLUMN col_name + | MODIFY COLUMN col_name column_type + | RENAME COLUMN old_col_name new_col_name +} + +alter_table_options: + alter_table_option ... + +alter_table_option: { + TTL value + | COMMENT 'string_value' +} -::: +``` -## 删除数据表 +**使用说明** +对普通表可以进行如下修改操作 +1. ADD COLUMN:添加列。 +2. DROP COLUMN:删除列。 +3. ODIFY COLUMN:修改列定义,如果数据列的类型是可变长类型,那么可以使用此指令修改其宽度,只能改大,不能改小。 +4. RENAME COLUMN:修改列名称。 -``` -DROP TABLE [IF EXISTS] tb_name; +### 增加列 + +```sql +ALTER TABLE tb_name ADD COLUMN field_name data_type; ``` -## 显示当前数据库下的所有数据表信息 +### 删除列 -``` -SHOW TABLES [LIKE tb_name_wildchar]; +```sql +ALTER TABLE tb_name DROP COLUMN field_name; ``` -显示当前数据库下的所有数据表信息。 - -## 显示一个数据表的创建语句 +### 修改列宽 +```sql +ALTER TABLE tb_name MODIFY COLUMN field_name data_type(length); ``` -SHOW CREATE TABLE tb_name; + +### 修改列名 + +```sql +ALTER TABLE tb_name RENAME COLUMN old_col_name new_col_name ``` -常用于数据库迁移。对一个已经存在的数据表,返回其创建语句;在另一个集群中执行该语句,就能得到一个结构完全相同的数据表。 +## 修改子表 + +ALTER TABLE [db_name.]tb_name alter_table_clause + +alter_table_clause: { + alter_table_options + | SET TAG tag_name = new_tag_value +} + +alter_table_options: + alter_table_option ... + +alter_table_option: { + TTL value + | COMMENT 'string_value' +} + +**使用说明** +1. 对子表的列和标签的修改,除了更改标签值以外,都要通过超级表才能进行。 -## 获取表的结构信息 +### 修改子表标签值 ``` -DESCRIBE tb_name; +ALTER TABLE tb_name SET TAG tag_name=new_tag_value; ``` -## 修改表定义 +## 删除表 -### 表增加列 +可以在一条SQL语句中删除一个或多个普通表或子表。 +```sql +DROP TABLE [IF EXISTS] [db_name.]tb_name [, [IF EXISTS] [db_name.]tb_name] ... ``` -ALTER TABLE tb_name ADD COLUMN field_name data_type; -``` - -:::info -1. 列的最大个数为 1024,最小个数为 2;(从 2.1.7.0 版本开始,改为最多允许 4096 列) -2. 列名最大长度为 64。 +## 查看表的信息 -::: +### 显示所有表 -### 表删除列 +如下SQL语句可以列出当前数据库中的所有表名。 +```sql +SHOW TABLES [LIKE tb_name_wildchar]; ``` -ALTER TABLE tb_name DROP COLUMN field_name; -``` - -如果表是通过超级表创建,更改表结构的操作只能对超级表进行。同时针对超级表的结构更改对所有通过该结构创建的表生效。对于不是通过超级表创建的表,可以直接修改表结构。 -### 表修改列宽 +### 显示表创建语句 ``` -ALTER TABLE tb_name MODIFY COLUMN field_name data_type(length); +SHOW CREATE TABLE tb_name; ``` -如果数据列的类型是可变长格式(BINARY 或 NCHAR),那么可以使用此指令修改其宽度(只能改大,不能改小)。(2.1.3.0 版本新增) -如果表是通过超级表创建,更改表结构的操作只能对超级表进行。同时针对超级表的结构更改对所有通过该结构创建的表生效。对于不是通过超级表创建的表,可以直接修改表结构。 +常用于数据库迁移。对一个已经存在的数据表,返回其创建语句;在另一个集群中执行该语句,就能得到一个结构完全相同的数据表。 -### 修改子表标签值 +### 获取表结构信息 ``` -ALTER TABLE tb_name SET TAG tag_name=new_tag_value; -``` - -如果表是通过超级表创建,可以使用此指令修改其标签值 +DESCRIBE tb_name; +``` \ No newline at end of file diff --git a/docs/zh/12-taos-sql/04-stable.md b/docs/zh/12-taos-sql/04-stable.md index 3901427736e80bc8dd0dd87b454947af6e586561..4d0f5e47650c744176bba51b1d2a4f3413bc055b 100644 --- a/docs/zh/12-taos-sql/04-stable.md +++ b/docs/zh/12-taos-sql/04-stable.md @@ -3,38 +3,31 @@ sidebar_label: 超级表管理 title: 超级表 STable 管理 --- -:::note - -在 2.0.15.0 及以后的版本中开始支持 STABLE 保留字。也即,在本节后文的指令说明中,CREATE、DROP、ALTER 三个指令在 2.0.15.0 之前的版本中 STABLE 保留字需写作 TABLE。 - -::: - ## 创建超级表 +```sql +CREATE STABLE [IF NOT EXISTS] stb_name (create_definition [, create_definitionn] ...) TAGS (create_definition [, create_definition] ...) [table_options] + +create_definition: + col_name column_definition + +column_definition: + type_name [COMMENT 'string_value'] ``` -CREATE STABLE [IF NOT EXISTS] stb_name (timestamp_field_name TIMESTAMP, field1_name data_type1 [, field2_name data_type2 ...]) TAGS (tag1_name tag_type1, tag2_name tag_type2 [, tag3_name tag_type3]); -``` - -创建 STable,与创建表的 SQL 语法相似,但需要指定 TAGS 字段的名称和类型。 -:::info - -1. TAGS 列的数据类型不能是 timestamp 类型;(从 2.1.3.0 版本开始,TAGS 列中支持使用 timestamp 类型,但需注意在 TAGS 中的 timestamp 列写入数据时需要提供给定值,而暂不支持四则运算,例如 `NOW + 10s` 这类表达式) -2. TAGS 列名不能与其他列名相同; -3. TAGS 列名不能为预留关键字(参见:[参数限制与保留关键字](/taos-sql/keywords/) 章节); -4. TAGS 最多允许 128 个,至少 1 个,总长度不超过 16 KB。 - -::: - -## 删除超级表 - -``` -DROP STABLE [IF EXISTS] stb_name; -``` +**使用说明** +- 超级表中列的最大个数为 4096,需要注意,这里的 4096 是包含 TAG 列在内的,最小个数为 3,包含一个时间戳主键、一个 TAG 列和一个数据列。 +- 建表时可以给列或标签附加注释。 +- TAGS语法指定超级表的标签列,标签列需要遵循以下约定: + - TAGS 中的 TIMESTAMP 列写入数据时需要提供给定值,而暂不支持四则运算,例如 NOW + 10s 这类表达式。 + - TAGS 列名不能与其他列名相同。 + - TAGS 列名不能为预留关键字。 + - TAGS 最多允许 128 个,至少 1 个,总长度不超过 16 KB。 +- 关于表参数的详细说明,参见 CREATE TABLE 中的介绍。 -删除 STable 会自动删除通过 STable 创建的子表。 +## 查看超级表 -## 显示当前数据库下的所有超级表信息 +### 显示当前数据库下的所有超级表信息 ``` SHOW STABLES [LIKE tb_name_wildcard]; @@ -42,7 +35,7 @@ SHOW STABLES [LIKE tb_name_wildcard]; 查看数据库内全部 STable,及其相关信息,包括 STable 的名称、创建时间、列数量、标签(TAG)数量、通过该 STable 建表的数量。 -## 显示一个超级表的创建语句 +### 显示一个超级表的创建语句 ``` SHOW CREATE STABLE stb_name; @@ -50,40 +43,81 @@ SHOW CREATE STABLE stb_name; 常用于数据库迁移。对一个已经存在的超级表,返回其创建语句;在另一个集群中执行该语句,就能得到一个结构完全相同的超级表。 -## 获取超级表的结构信息 +### 获取超级表的结构信息 ``` DESCRIBE stb_name; ``` -## 修改超级表普通列 - -### 超级表增加列 +## 删除超级表 ``` -ALTER STABLE stb_name ADD COLUMN field_name data_type; +DROP STABLE [IF EXISTS] [db_name.]stb_name +``` + +删除 STable 会自动删除通过 STable 创建的子表以及子表中的所有数据。 + +## 修改超级表 + +```sql +ALTER STABLE [db_name.]tb_name alter_table_clause + +alter_table_clause: { + alter_table_options + | ADD COLUMN col_name column_type + | DROP COLUMN col_name + | MODIFY COLUMN col_name column_type + | ADD TAG tag_name tag_type + | DROP TAG tag_name + | MODIFY TAG tag_name tag_type + | RENAME TAG old_tag_name new_tag_name +} + +alter_table_options: + alter_table_option ... + +alter_table_option: { + COMMENT 'string_value' +} + ``` -### 超级表删除列 +**使用说明** + +修改超级表的结构会对其下的所有子表生效。无法针对某个特定子表修改表结构。标签结构的修改需要对超级表下发,TDengine 会自动作用于此超级表的所有子表。 + +- ADD COLUMN:添加列。 +- DROP COLUMN:删除列。 +- MODIFY COLUMN:修改列定义,如果数据列的类型是可变长类型,那么可以使用此指令修改其宽度,只能改大,不能改小。 +- ADD TAG:给超级表添加一个标签。 +- DROP TAG:删除超级表的一个标签。从超级表删除某个标签后,该超级表下的所有子表也会自动删除该标签。 +- MODIFY TAG:修改超级表的一个标签的定义。如果标签的类型是可变长类型,那么可以使用此指令修改其宽度,只能改大,不能改小。 +- RENAME TAG:修改超级表的一个标签的名称。从超级表修改某个标签名后,该超级表下的所有子表也会自动更新该标签名。 + +### 增加列 ``` -ALTER STABLE stb_name DROP COLUMN field_name; +ALTER STABLE stb_name ADD COLUMN col_name column_type; ``` -### 超级表修改列宽 +### 删除列 ``` -ALTER STABLE stb_name MODIFY COLUMN field_name data_type(length); +ALTER STABLE stb_name DROP COLUMN col_name; ``` -如果数据列的类型是可变长格式(BINARY 或 NCHAR),那么可以使用此指令修改其宽度(只能改大,不能改小)。(2.1.3.0 版本新增) +### 修改列宽 + +``` +ALTER STABLE stb_name MODIFY COLUMN col_name data_type(length); +``` -## 修改超级表标签列 +如果数据列的类型是可变长格式(BINARY 或 NCHAR),那么可以使用此指令修改其宽度(只能改大,不能改小)。 ### 添加标签 ``` -ALTER STABLE stb_name ADD TAG new_tag_name tag_type; +ALTER STABLE stb_name ADD TAG tag_name tag_type; ``` 为 STable 增加一个新的标签,并指定新标签的类型。标签总数不能超过 128 个,总长度不超过 16KB 。 @@ -99,7 +133,7 @@ ALTER STABLE stb_name DROP TAG tag_name; ### 修改标签名 ``` -ALTER STABLE stb_name CHANGE TAG old_tag_name new_tag_name; +ALTER STABLE stb_name RENAME TAG old_tag_name new_tag_name; ``` 修改超级表的标签名,从超级表修改某个标签名后,该超级表下的所有子表也会自动更新该标签名。 diff --git a/include/common/tmsg.h b/include/common/tmsg.h index 0d0eb841bc31c4314e07ec2ebc3b5154922f8b17..eb12bb407386aee2456548005950ae3e1ca6785f 100644 --- a/include/common/tmsg.h +++ b/include/common/tmsg.h @@ -1154,6 +1154,10 @@ typedef struct { int32_t numOfRetensions; SArray* pRetensions; // SRetention void* pTsma; + int32_t walRetentionPeriod; + int64_t walRetentionSize; + int32_t walRollPeriod; + int64_t walSegmentSize; } SCreateVnodeReq; int32_t tSerializeSCreateVnodeReq(void* buf, int32_t bufLen, SCreateVnodeReq* pReq); diff --git a/include/libs/function/function.h b/include/libs/function/function.h index 8cb48cc9f05df87c1636ed60c032ddadeac75d8e..8fa63bbd4502a121b0e26a29df948d8f34f77ef1 100644 --- a/include/libs/function/function.h +++ b/include/libs/function/function.h @@ -143,6 +143,7 @@ typedef struct SqlFunctionCtx { struct SExprInfo *pExpr; struct SDiskbasedBuf *pBuf; struct SSDataBlock *pSrcBlock; + struct SSDataBlock *pDstBlock; // used by indifinite rows function to set selectivity int32_t curBufPage; bool increase; diff --git a/include/libs/wal/wal.h b/include/libs/wal/wal.h index 220c4f73e0867b9ca71ef4adb2a3c9d5ac9ffa21..5b8d70fb7c127d476566f54623c43783cd5a240a 100644 --- a/include/libs/wal/wal.h +++ b/include/libs/wal/wal.h @@ -103,8 +103,8 @@ typedef struct SWal { int32_t fsyncSeq; // meta SWalVer vers; - TdFilePtr pWriteLogTFile; - TdFilePtr pWriteIdxTFile; + TdFilePtr pLogFile; + TdFilePtr pIdxFile; int32_t writeCur; SArray *fileInfoSet; // SArray // status diff --git a/source/client/src/clientImpl.c b/source/client/src/clientImpl.c index 03238e6747c71bae8c9286a9ad87010a21c4d843..eecb22abe5ee4aa4162791334327b93919866b3e 100644 --- a/source/client/src/clientImpl.c +++ b/source/client/src/clientImpl.c @@ -2019,7 +2019,7 @@ int32_t transferTableNameList(const char* tbList, int32_t acctId, char* dbName, } if (('a' <= *(tbList + i) && 'z' >= *(tbList + i)) || ('A' <= *(tbList + i) && 'Z' >= *(tbList + i)) || - ('0' <= *(tbList + i) && '9' >= *(tbList + i))) { + ('0' <= *(tbList + i) && '9' >= *(tbList + i)) || ('_' == *(tbList + i))) { if (vLen[vIdx] > 0) { goto _return; } diff --git a/source/client/src/clientMain.c b/source/client/src/clientMain.c index 77b31011a3d0109045154d72cda8bc1a4af65f6a..416d50d987c27cfc5a312a679ed353b3a0c8dc26 100644 --- a/source/client/src/clientMain.c +++ b/source/client/src/clientMain.c @@ -973,7 +973,7 @@ int taos_load_table_info(TAOS *taos, const char *tableNameList) { conn.mgmtEps = getEpSet_s(&pTscObj->pAppInfo->mgmtEp); - code = catalogAsyncGetAllMeta(pCtg, &conn, &catalogReq, syncCatalogFn, NULL, NULL); + code = catalogAsyncGetAllMeta(pCtg, &conn, &catalogReq, syncCatalogFn, pRequest->body.param, NULL); if (code) { goto _return; } diff --git a/source/common/src/tdatablock.c b/source/common/src/tdatablock.c index f0b5de9838ad05fd16db54f22c4dd13f07595988..e516bddac164034bf37fb43384a96b61db4fedfb 100644 --- a/source/common/src/tdatablock.c +++ b/source/common/src/tdatablock.c @@ -1763,9 +1763,9 @@ char* dumpBlockData(SSDataBlock* pDataBlock, const char* flag, char** pDataBuf) int32_t colNum = taosArrayGetSize(pDataBlock->pDataBlock); int32_t rows = pDataBlock->info.rows; int32_t len = 0; - len += snprintf(dumpBuf + len, size - len, "===stream===%s |block type %d |child id %d|group id:%" PRIu64 "| uid:%ld|\n", flag, + len += snprintf(dumpBuf + len, size - len, "===stream===%s |block type %d|child id %d|group id:%" PRIu64 "|uid:%ld|rows:%d\n", flag, (int32_t)pDataBlock->info.type, pDataBlock->info.childId, pDataBlock->info.groupId, - pDataBlock->info.uid); + pDataBlock->info.uid, pDataBlock->info.rows); if (len >= size - 1) return dumpBuf; for (int32_t j = 0; j < rows; j++) { diff --git a/source/common/src/tmsg.c b/source/common/src/tmsg.c index f87d336ec2107d83cea45b615770039f235d0852..6bdb6a4e655acfa177d45db716f1eb67556d3515 100644 --- a/source/common/src/tmsg.c +++ b/source/common/src/tmsg.c @@ -3750,6 +3750,10 @@ int32_t tSerializeSCreateVnodeReq(void *buf, int32_t bufLen, SCreateVnodeReq *pR uint32_t tsmaLen = (uint32_t)(htonl(((SMsgHead *)pReq->pTsma)->contLen)); if (tEncodeBinary(&encoder, (const uint8_t *)pReq->pTsma, tsmaLen) < 0) return -1; } + if (tEncodeI32(&encoder, pReq->walRetentionPeriod) < 0) return -1; + if (tEncodeI64(&encoder, pReq->walRetentionSize) < 0) return -1; + if (tEncodeI32(&encoder, pReq->walRollPeriod) < 0) return -1; + if (tEncodeI64(&encoder, pReq->walSegmentSize) < 0) return -1; tEndEncode(&encoder); @@ -3818,6 +3822,11 @@ int32_t tDeserializeSCreateVnodeReq(void *buf, int32_t bufLen, SCreateVnodeReq * if (tDecodeBinary(&decoder, (uint8_t **)&pReq->pTsma, NULL) < 0) return -1; } + if (tDecodeI32(&decoder, &pReq->walRetentionPeriod) < 0) return -1; + if (tDecodeI64(&decoder, &pReq->walRetentionSize) < 0) return -1; + if (tDecodeI32(&decoder, &pReq->walRollPeriod) < 0) return -1; + if (tDecodeI64(&decoder, &pReq->walSegmentSize) < 0) return -1; + tEndDecode(&decoder); tDecoderClear(&decoder); return 0; diff --git a/source/dnode/mgmt/mgmt_vnode/src/vmHandle.c b/source/dnode/mgmt/mgmt_vnode/src/vmHandle.c index 0471e2b85043e0286909d3174e8c937dd4e22f08..cb061e6d1c4a3a2148996cc781b437f8af0fcb9f 100644 --- a/source/dnode/mgmt/mgmt_vnode/src/vmHandle.c +++ b/source/dnode/mgmt/mgmt_vnode/src/vmHandle.c @@ -160,6 +160,13 @@ static void vmGenerateVnodeCfg(SCreateVnodeReq *pCreate, SVnodeCfg *pCfg) { } pCfg->walCfg.vgId = pCreate->vgId; + pCfg->walCfg.fsyncPeriod = pCreate->fsyncPeriod; + pCfg->walCfg.retentionPeriod = pCreate->walRetentionPeriod; + pCfg->walCfg.rollPeriod = pCreate->walRollPeriod; + pCfg->walCfg.retentionSize = pCreate->walRetentionSize; + pCfg->walCfg.segSize = pCreate->walSegmentSize; + pCfg->walCfg.level = pCreate->walLevel; + pCfg->hashBegin = pCreate->hashBegin; pCfg->hashEnd = pCreate->hashEnd; pCfg->hashMethod = pCreate->hashMethod; diff --git a/source/dnode/mnode/impl/inc/mndDef.h b/source/dnode/mnode/impl/inc/mndDef.h index c9997fa3d5860e92b6b92c3d33d35c32080be866..0ff9b4102d5623efd8b406367b282fda150d1190 100644 --- a/source/dnode/mnode/impl/inc/mndDef.h +++ b/source/dnode/mnode/impl/inc/mndDef.h @@ -302,9 +302,13 @@ typedef struct { int8_t strict; int8_t hashMethod; // default is 1 int8_t cacheLast; + int8_t schemaless; int32_t numOfRetensions; SArray* pRetensions; - int8_t schemaless; + int32_t walRetentionPeriod; + int64_t walRetentionSize; + int32_t walRollPeriod; + int64_t walSegmentSize; } SDbCfg; typedef struct { diff --git a/source/dnode/mnode/impl/src/mndDb.c b/source/dnode/mnode/impl/src/mndDb.c index 064ef9b40a772790ff01284cee9103dfe642f78b..86787fcd012bfc6bec3448940940e485e630f404 100644 --- a/source/dnode/mnode/impl/src/mndDb.c +++ b/source/dnode/mnode/impl/src/mndDb.c @@ -120,6 +120,10 @@ static SSdbRaw *mndDbActionEncode(SDbObj *pDb) { SDB_SET_INT8(pRaw, dataPos, pRetension->keepUnit, _OVER) } SDB_SET_INT8(pRaw, dataPos, pDb->cfg.schemaless, _OVER) + SDB_SET_INT32(pRaw, dataPos, pDb->cfg.walRetentionPeriod, _OVER) + SDB_SET_INT64(pRaw, dataPos, pDb->cfg.walRetentionSize, _OVER) + SDB_SET_INT32(pRaw, dataPos, pDb->cfg.walRollPeriod, _OVER) + SDB_SET_INT64(pRaw, dataPos, pDb->cfg.walSegmentSize, _OVER) SDB_SET_RESERVE(pRaw, dataPos, DB_RESERVE_SIZE, _OVER) SDB_SET_DATALEN(pRaw, dataPos, _OVER) @@ -199,6 +203,10 @@ static SSdbRow *mndDbActionDecode(SSdbRaw *pRaw) { } } SDB_GET_INT8(pRaw, dataPos, &pDb->cfg.schemaless, _OVER) + SDB_GET_INT32(pRaw, dataPos, &pDb->cfg.walRetentionPeriod, _OVER) + SDB_GET_INT64(pRaw, dataPos, &pDb->cfg.walRetentionSize, _OVER) + SDB_GET_INT32(pRaw, dataPos, &pDb->cfg.walRollPeriod, _OVER) + SDB_GET_INT64(pRaw, dataPos, &pDb->cfg.walSegmentSize, _OVER) SDB_GET_RESERVE(pRaw, dataPos, DB_RESERVE_SIZE, _OVER) taosInitRWLatch(&pDb->lock); @@ -318,6 +326,10 @@ static int32_t mndCheckDbCfg(SMnode *pMnode, SDbCfg *pCfg) { terrno = TSDB_CODE_MND_NO_ENOUGH_DNODES; return -1; } + if (pCfg->walRetentionPeriod < TSDB_DB_MIN_WAL_RETENTION_PERIOD) return -1; + if (pCfg->walRetentionSize < TSDB_DB_MIN_WAL_RETENTION_SIZE) return -1; + if (pCfg->walRollPeriod < TSDB_DB_MIN_WAL_ROLL_PERIOD) return -1; + if (pCfg->walSegmentSize < TSDB_DB_MIN_WAL_SEGMENT_SIZE) return -1; terrno = 0; return terrno; @@ -345,6 +357,12 @@ static void mndSetDefaultDbCfg(SDbCfg *pCfg) { if (pCfg->cacheLastSize <= 0) pCfg->cacheLastSize = TSDB_DEFAULT_CACHE_SIZE; if (pCfg->numOfRetensions < 0) pCfg->numOfRetensions = 0; if (pCfg->schemaless < 0) pCfg->schemaless = TSDB_DB_SCHEMALESS_OFF; + if (pCfg->walRetentionPeriod < 0 && pCfg->walRetentionPeriod != -1) + pCfg->walRetentionPeriod = TSDB_DEFAULT_DB_WAL_RETENTION_PERIOD; + if (pCfg->walRetentionSize < 0 && pCfg->walRetentionSize != -1) + pCfg->walRetentionSize = TSDB_DEFAULT_DB_WAL_RETENTION_SIZE; + if (pCfg->walRollPeriod < 0) pCfg->walRollPeriod = TSDB_DEFAULT_DB_WAL_ROLL_PERIOD; + if (pCfg->walSegmentSize < 0) pCfg->walSegmentSize = TSDB_DEFAULT_DB_WAL_SEGMENT_SIZE; } static int32_t mndSetCreateDbRedoLogs(SMnode *pMnode, STrans *pTrans, SDbObj *pDb, SVgObj *pVgroups) { @@ -457,6 +475,10 @@ static int32_t mndCreateDb(SMnode *pMnode, SRpcMsg *pReq, SCreateDbReq *pCreate, .cacheLast = pCreate->cacheLast, .hashMethod = 1, .schemaless = pCreate->schemaless, + .walRetentionPeriod = pCreate->walRetentionPeriod, + .walRetentionSize = pCreate->walRetentionSize, + .walRollPeriod = pCreate->walRollPeriod, + .walSegmentSize = pCreate->walSegmentSize, }; dbObj.cfg.numOfRetensions = pCreate->numOfRetensions; diff --git a/source/dnode/mnode/impl/src/mndVgroup.c b/source/dnode/mnode/impl/src/mndVgroup.c index 3eb3a6cd1fd1f3423fe50829d307a1dfea815272..4625b2ab01544c3970e8509aa8b992e725812a45 100644 --- a/source/dnode/mnode/impl/src/mndVgroup.c +++ b/source/dnode/mnode/impl/src/mndVgroup.c @@ -230,6 +230,10 @@ void *mndBuildCreateVnodeReq(SMnode *pMnode, SDnodeObj *pDnode, SDbObj *pDb, SVg createReq.standby = standby; createReq.isTsma = pVgroup->isTsma; createReq.pTsma = pVgroup->pTsma; + createReq.walRetentionPeriod = pDb->cfg.walRetentionPeriod; + createReq.walRetentionSize = pDb->cfg.walRetentionSize; + createReq.walRollPeriod = pDb->cfg.walRollPeriod; + createReq.walSegmentSize = pDb->cfg.walSegmentSize; for (int32_t v = 0; v < pVgroup->replica; ++v) { SReplica *pReplica = &createReq.replicas[v]; diff --git a/source/dnode/vnode/src/inc/tsdb.h b/source/dnode/vnode/src/inc/tsdb.h index 04bf6bcc2b50ccf1a02ace316234acb78041d446..d8c84e952b43a5c0f0cfa64058d8f4a6fb77ca3e 100644 --- a/source/dnode/vnode/src/inc/tsdb.h +++ b/source/dnode/vnode/src/inc/tsdb.h @@ -371,8 +371,8 @@ struct SBlockIdx { struct SMapData { int32_t nItem; - int32_t *aOffset; int32_t nData; + int32_t *aOffset; uint8_t *pData; }; diff --git a/source/dnode/vnode/src/meta/metaOpen.c b/source/dnode/vnode/src/meta/metaOpen.c index 59df35d5542585c6d9f24ae2f64d2b1dfc48e42e..7c7d14e33777a3e2b9905e6309c1ba12bbb6f40e 100644 --- a/source/dnode/vnode/src/meta/metaOpen.c +++ b/source/dnode/vnode/src/meta/metaOpen.c @@ -180,11 +180,41 @@ int metaClose(SMeta *pMeta) { return 0; } -int32_t metaRLock(SMeta *pMeta) { return taosThreadRwlockRdlock(&pMeta->lock); } +int32_t metaRLock(SMeta *pMeta) { + int32_t ret = 0; -int32_t metaWLock(SMeta *pMeta) { return taosThreadRwlockWrlock(&pMeta->lock); } + metaDebug("meta rlock %p B", &pMeta->lock); -int32_t metaULock(SMeta *pMeta) { return taosThreadRwlockUnlock(&pMeta->lock); } + ret = taosThreadRwlockRdlock(&pMeta->lock); + + metaDebug("meta rlock %p E", &pMeta->lock); + + return ret; +} + +int32_t metaWLock(SMeta *pMeta) { + int32_t ret = 0; + + metaDebug("meta wlock %p B", &pMeta->lock); + + ret = taosThreadRwlockWrlock(&pMeta->lock); + + metaDebug("meta wlock %p E", &pMeta->lock); + + return ret; +} + +int32_t metaULock(SMeta *pMeta) { + int32_t ret = 0; + + metaDebug("meta ulock %p B", &pMeta->lock); + + ret = taosThreadRwlockUnlock(&pMeta->lock); + + metaDebug("meta ulock %p E", &pMeta->lock); + + return ret; +} static int tbDbKeyCmpr(const void *pKey1, int kLen1, const void *pKey2, int kLen2) { STbDbKey *pTbDbKey1 = (STbDbKey *)pKey1; @@ -259,7 +289,7 @@ static int ctbIdxKeyCmpr(const void *pKey1, int kLen1, const void *pKey2, int kL static int tagIdxKeyCmpr(const void *pKey1, int kLen1, const void *pKey2, int kLen2) { STagIdxKey *pTagIdxKey1 = (STagIdxKey *)pKey1; STagIdxKey *pTagIdxKey2 = (STagIdxKey *)pKey2; - tb_uid_t uid1, uid2; + tb_uid_t uid1 = 0, uid2 = 0; int c; // compare suid @@ -287,14 +317,15 @@ static int tagIdxKeyCmpr(const void *pKey1, int kLen1, const void *pKey2, int kL // all not NULL, compr tag vals c = doCompare(pTagIdxKey1->data, pTagIdxKey2->data, pTagIdxKey1->type, 0); if (c) return c; + } - if (IS_VAR_DATA_TYPE(pTagIdxKey1->type)) { - uid1 = *(tb_uid_t *)(pTagIdxKey1->data + varDataTLen(pTagIdxKey1->data)); - uid2 = *(tb_uid_t *)(pTagIdxKey2->data + varDataTLen(pTagIdxKey2->data)); - } else { - uid1 = *(tb_uid_t *)(pTagIdxKey1->data + tDataTypes[pTagIdxKey1->type].bytes); - uid2 = *(tb_uid_t *)(pTagIdxKey2->data + tDataTypes[pTagIdxKey2->type].bytes); - } + // both null or tag values are equal, then continue to compare uids + if (IS_VAR_DATA_TYPE(pTagIdxKey1->type)) { + uid1 = *(tb_uid_t *)(pTagIdxKey1->data + varDataTLen(pTagIdxKey1->data)); + uid2 = *(tb_uid_t *)(pTagIdxKey2->data + varDataTLen(pTagIdxKey2->data)); + } else { + uid1 = *(tb_uid_t *)(pTagIdxKey1->data + tDataTypes[pTagIdxKey1->type].bytes); + uid2 = *(tb_uid_t *)(pTagIdxKey2->data + tDataTypes[pTagIdxKey2->type].bytes); } // compare uid diff --git a/source/dnode/vnode/src/tq/tq.c b/source/dnode/vnode/src/tq/tq.c index 118e3a5d43d159abf7f3ed8a901c2707e7987cdf..2adfc92ab1f7f0f88ae9dd4f14147c278b3fefdb 100644 --- a/source/dnode/vnode/src/tq/tq.c +++ b/source/dnode/vnode/src/tq/tq.c @@ -583,7 +583,7 @@ int32_t tqProcessVgChangeReq(STQ* pTq, char* msg, int32_t msgLen) { pHandle->execHandle.execTb.suid = req.suid; SArray* tbUidList = taosArrayInit(0, sizeof(int64_t)); vnodeGetCtbIdList(pTq->pVnode, req.suid, tbUidList); - tqDebug("vgId:%d, tq try get suid:%" PRId64, pTq->pVnode->config.vgId, req.suid); + tqDebug("vgId:%d, tq try to get all ctb, suid:%" PRId64, pTq->pVnode->config.vgId, req.suid); for (int32_t i = 0; i < taosArrayGetSize(tbUidList); i++) { int64_t tbUid = *(int64_t*)taosArrayGet(tbUidList, i); tqDebug("vgId:%d, idx %d, uid:%" PRId64, TD_VID(pTq->pVnode), i, tbUid); diff --git a/source/dnode/vnode/src/tsdb/tsdbRead.c b/source/dnode/vnode/src/tsdb/tsdbRead.c index ea8ac09429af5018e4b01b3bc40e9a3242295fa9..1b162bf4888b9da95884aca01a014157c723bad1 100644 --- a/source/dnode/vnode/src/tsdb/tsdbRead.c +++ b/source/dnode/vnode/src/tsdb/tsdbRead.c @@ -31,7 +31,7 @@ typedef struct { typedef struct STableBlockScanInfo { uint64_t uid; TSKEY lastKey; - SBlockIdx blockIdx; + SMapData mapData; // block info (compressed) SArray* pBlockList; // block data index list SIterInfo iter; // mem buffer skip list iterator SIterInfo iiter; // imem buffer skip list iterator @@ -42,7 +42,7 @@ typedef struct STableBlockScanInfo { typedef struct SBlockOrderWrapper { int64_t uid; - SBlock* pBlock; + int64_t offset; } SBlockOrderWrapper; typedef struct SBlockOrderSupporter { @@ -53,11 +53,13 @@ typedef struct SBlockOrderSupporter { } SBlockOrderSupporter; typedef struct SIOCostSummary { - int64_t blockLoadTime; - int64_t smaLoadTime; - int64_t checkForNextTime; + int64_t numOfBlocks; + double blockLoadTime; + double buildmemBlock; int64_t headFileLoad; - int64_t headFileLoadTime; + double headFileLoadTime; + int64_t smaData; + double smaLoadTime; } SIOCostSummary; typedef struct SBlockLoadSuppInfo { @@ -86,6 +88,8 @@ typedef struct SDataBlockIter { int32_t index; SArray* blockList; // SArray int32_t order; + SBlock block; // current SBlock data + SHashObj* pTableMap; } SDataBlockIter; typedef struct SFileBlockDumpInfo { @@ -183,7 +187,7 @@ static int32_t setColumnIdSlotList(STsdbReader* pReader, SSDataBlock* pBlock) { static SHashObj* createDataBlockScanInfo(STsdbReader* pTsdbReader, const STableKeyInfo* idList, int32_t numOfTables) { // allocate buffer in order to load data blocks from file - // todo use simple hash instead + // todo use simple hash instead, optimize the memory consumption SHashObj* pTableMap = taosHashInit(numOfTables, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BIGINT), false, HASH_NO_LOCK); if (pTableMap == NULL) { @@ -244,6 +248,7 @@ static void destroyBlockScanInfo(SHashObj* pTableMap) { p->delSkyline = taosArrayDestroy(p->delSkyline); p->pBlockList = taosArrayDestroy(p->pBlockList); + tMapDataClear(&p->mapData); } taosHashCleanup(pTableMap); @@ -320,6 +325,8 @@ static bool filesetIteratorNext(SFilesetIter* pIter, STsdbReader* pReader) { goto _err; } + pReader->cost.headFileLoad += 1; + int32_t fid = pReader->status.pCurrentFileset->fid; tsdbFidKeyRange(fid, pReader->pTsdb->keepCfg.days, pReader->pTsdb->keepCfg.precision, &win.skey, &win.ekey); @@ -347,7 +354,7 @@ _err: return false; } -static void resetDataBlockIterator(SDataBlockIter* pIter, int32_t order) { +static void resetDataBlockIterator(SDataBlockIter* pIter, int32_t order, SHashObj* pTableMap) { pIter->order = order; pIter->index = -1; pIter->numOfBlocks = -1; @@ -356,6 +363,7 @@ static void resetDataBlockIterator(SDataBlockIter* pIter, int32_t order) { } else { taosArrayClear(pIter->blockList); } + pIter->pTableMap = pTableMap; } static void cleanupDataBlockIterator(SDataBlockIter* pIter) { taosArrayDestroy(pIter->blockList); } @@ -521,7 +529,7 @@ _end: // } static int32_t doLoadBlockIndex(STsdbReader* pReader, SDataFReader* pFileReader, SArray* pIndexList) { - SArray* aBlockIdx = taosArrayInit(0, sizeof(SBlockIdx)); + SArray* aBlockIdx = taosArrayInit(8, sizeof(SBlockIdx)); int64_t st = taosGetTimestampUs(); int32_t code = tsdbReadBlockIdx(pFileReader, aBlockIdx, NULL); @@ -554,16 +562,18 @@ static int32_t doLoadBlockIndex(STsdbReader* pReader, SDataFReader* pFileReader, STableBlockScanInfo* pScanInfo = p; if (pScanInfo->pBlockList == NULL) { - pScanInfo->pBlockList = taosArrayInit(16, sizeof(SBlock)); + pScanInfo->pBlockList = taosArrayInit(4, sizeof(int32_t)); } - pScanInfo->blockIdx = *pBlockIdx; taosArrayPush(pIndexList, pBlockIdx); } int64_t et2 = taosGetTimestampUs(); - tsdbDebug("load block index for %d tables completed, elapsed time:%.2f ms, set blockIdx:%.2f ms, size:%d bytes %s", - (int32_t)num, (et1 - st)/1000.0, (et2-et1)/1000.0, num * sizeof(SBlockIdx), pReader->idStr); + tsdbDebug("load block index for %d tables completed, elapsed time:%.2f ms, set blockIdx:%.2f ms, size:%.2f Kb %s", + (int32_t)num, (et1 - st)/1000.0, (et2-et1)/1000.0, num * sizeof(SBlockIdx)/1024.0, pReader->idStr); + + pReader->cost.headFileLoadTime += (et1 - st) / 1000.0; + _end: taosArrayDestroy(aBlockIdx); return code; @@ -584,23 +594,22 @@ static int32_t doLoadFileBlock(STsdbReader* pReader, SArray* pIndexList, uint32_ break; } + tMapDataClear(&px->mapData); taosArrayClear(px->pBlockList); } for (int32_t i = 0; i < numOfTables; ++i) { SBlockIdx* pBlockIdx = taosArrayGet(pIndexList, i); - SMapData mapData = {0}; - tMapDataReset(&mapData); - tsdbReadBlock(pReader->pFileReader, pBlockIdx, &mapData, NULL); + STableBlockScanInfo* pScanInfo = taosHashGet(pReader->status.pTableMap, &pBlockIdx->uid, sizeof(int64_t)); - size += mapData.nData; + tMapDataReset(&pScanInfo->mapData); + tsdbReadBlock(pReader->pFileReader, pBlockIdx, &pScanInfo->mapData, NULL); - STableBlockScanInfo* pScanInfo = taosHashGet(pReader->status.pTableMap, &pBlockIdx->uid, sizeof(int64_t)); - for (int32_t j = 0; j < mapData.nItem; ++j) { + size += pScanInfo->mapData.nData; + for (int32_t j = 0; j < pScanInfo->mapData.nItem; ++j) { SBlock block = {0}; - - tMapDataGetItemByIdx(&mapData, j, &block, tGetBlock); + tMapDataGetItemByIdx(&pScanInfo->mapData, j, &block, tGetBlock); // 1. time range check if (block.minKey.ts > pReader->window.ekey || block.maxKey.ts < pReader->window.skey) { @@ -612,24 +621,26 @@ static int32_t doLoadFileBlock(STsdbReader* pReader, SArray* pIndexList, uint32_ continue; } - void* p = taosArrayPush(pScanInfo->pBlockList, &block); + void* p = taosArrayPush(pScanInfo->pBlockList, &j); if (p == NULL) { - tMapDataClear(&mapData); + tMapDataClear(&pScanInfo->mapData); return TSDB_CODE_OUT_OF_MEMORY; } (*numOfBlocks) += 1; } - tMapDataClear(&mapData); if (pScanInfo->pBlockList != NULL && taosArrayGetSize(pScanInfo->pBlockList) > 0) { (*numOfValidTables) += 1; } } - int64_t et = taosGetTimestampUs(); + double el = (taosGetTimestampUs() - st)/1000.0; tsdbDebug("load block of %d tables completed, blocks:%d in %d tables, size:%.2f Kb, elapsed time:%.2f ms %s", - numOfTables, *numOfBlocks, *numOfValidTables, size/1000.0, (et-st)/1000.0, pReader->idStr); + numOfTables, *numOfBlocks, *numOfValidTables, size/1000.0, el, pReader->idStr); + + pReader->cost.numOfBlocks += (*numOfBlocks); + pReader->cost.headFileLoadTime += el; return TSDB_CODE_SUCCESS; } @@ -657,13 +668,22 @@ static void doCopyColVal(SColumnInfoData* pColInfoData, int32_t rowIndex, int32_ } } +static SFileDataBlockInfo* getCurrentBlockInfo(SDataBlockIter* pBlockIter) { + SFileDataBlockInfo* pFBlockInfo = taosArrayGet(pBlockIter->blockList, pBlockIter->index); + return pFBlockInfo; +} + +static SBlock* getCurrentBlock(SDataBlockIter* pBlockIter) { + return &pBlockIter->block; +} + static int32_t copyBlockDataToSDataBlock(STsdbReader* pReader, STableBlockScanInfo* pBlockScanInfo) { SReaderStatus* pStatus = &pReader->status; SDataBlockIter* pBlockIter = &pStatus->blockIter; SBlockData* pBlockData = &pStatus->fileBlockData; SFileDataBlockInfo* pFBlock = getCurrentBlockInfo(pBlockIter); - SBlock* pBlock = taosArrayGet(pBlockScanInfo->pBlockList, pFBlock->tbBlockIdx); + SBlock* pBlock = getCurrentBlock(pBlockIter); SSDataBlock* pResBlock = pReader->pResBlock; int32_t numOfCols = blockDataGetNumOfCols(pResBlock); @@ -729,12 +749,12 @@ static int32_t copyBlockDataToSDataBlock(STsdbReader* pReader, STableBlockScanIn setBlockAllDumped(pDumpInfo, pBlock, pReader->order); - int64_t elapsedTime = (taosGetTimestampUs() - st); + double elapsedTime = (taosGetTimestampUs() - st) / 1000.0; pReader->cost.blockLoadTime += elapsedTime; int32_t unDumpedRows = asc ? pBlock->nRow - pDumpInfo->rowIndex : pDumpInfo->rowIndex + 1; tsdbDebug("%p load file block into buffer, global index:%d, table index:%d, brange:%" PRId64 "-%" PRId64 - ", rows:%d, remain:%d, minVer:%" PRId64 ", maxVer:%" PRId64 ", elapsed time:%" PRId64 " us, %s", + ", rows:%d, remain:%d, minVer:%" PRId64 ", maxVer:%" PRId64 ", elapsed time:%.2f ms, %s", pReader, pBlockIter->index, pFBlock->tbBlockIdx, pBlock->minKey.ts, pBlock->maxKey.ts, remain, unDumpedRows, pBlock->minVersion, pBlock->maxVersion, elapsedTime, pReader->idStr); @@ -746,27 +766,30 @@ static int32_t doLoadFileBlockData(STsdbReader* pReader, SDataBlockIter* pBlockI int64_t st = taosGetTimestampUs(); SFileDataBlockInfo* pFBlock = getCurrentBlockInfo(pBlockIter); - SBlock* pBlock = taosArrayGet(pBlockScanInfo->pBlockList, pFBlock->tbBlockIdx); + SBlock* pBlock = getCurrentBlock(pBlockIter); + SSDataBlock* pResBlock = pReader->pResBlock; int32_t numOfCols = blockDataGetNumOfCols(pResBlock); SBlockLoadSuppInfo* pSupInfo = &pReader->suppInfo; SFileBlockDumpInfo* pDumpInfo = &pReader->status.fBlockDumpInfo; - int32_t code = tsdbReadColData(pReader->pFileReader, &pBlockScanInfo->blockIdx, pBlock, pSupInfo->colIds, numOfCols, + SBlockIdx blockIdx = {.suid = pReader->suid, .uid = pBlockScanInfo->uid}; + int32_t code = tsdbReadColData(pReader->pFileReader, &blockIdx, pBlock, pSupInfo->colIds, numOfCols, pBlockData, NULL, NULL); if (code != TSDB_CODE_SUCCESS) { goto _error; } - int64_t elapsedTime = (taosGetTimestampUs() - st); + double elapsedTime = (taosGetTimestampUs() - st) / 1000.0; pReader->cost.blockLoadTime += elapsedTime; pDumpInfo->allDumped = false; tsdbDebug("%p load file block into buffer, global index:%d, table index:%d, brange:%" PRId64 "-%" PRId64 - ", rows:%d, minVer:%" PRId64 ", maxVer:%" PRId64 ", elapsed time:%" PRId64 " us, %s", + ", rows:%d, minVer:%" PRId64 ", maxVer:%" PRId64 ", elapsed time:%.2f ms, %s", pReader, pBlockIter->index, pFBlock->tbBlockIdx, pBlock->minKey.ts, pBlock->maxKey.ts, pBlock->nRow, pBlock->minVersion, pBlock->maxVersion, elapsedTime, pReader->idStr); + return TSDB_CODE_SUCCESS; _error: @@ -824,7 +847,21 @@ static int32_t fileDataBlockOrderCompar(const void* pLeft, const void* pRight, v SBlockOrderWrapper* pLeftBlock = &pSupporter->pDataBlockInfo[leftIndex][leftTableBlockIndex]; SBlockOrderWrapper* pRightBlock = &pSupporter->pDataBlockInfo[rightIndex][rightTableBlockIndex]; - return pLeftBlock->pBlock->aSubBlock[0].offset > pRightBlock->pBlock->aSubBlock[0].offset ? 1 : -1; + return pLeftBlock->offset > pRightBlock->offset ? 1 : -1; +} + +static int32_t doSetCurrentBlock(SDataBlockIter* pBlockIter) { + SFileDataBlockInfo* pFBlock = getCurrentBlockInfo(pBlockIter); + STableBlockScanInfo* pScanInfo = taosHashGet(pBlockIter->pTableMap, &pFBlock->uid, sizeof(pFBlock->uid)); + + int32_t* mapDataIndex = taosArrayGet(pScanInfo->pBlockList, pFBlock->tbBlockIdx); + tMapDataGetItemByIdx(&pScanInfo->mapData, *mapDataIndex, &pBlockIter->block, tGetBlock); + +#if 0 + qDebug("check file block, table uid:%"PRIu64" index:%d offset:%"PRId64", ", pScanInfo->uid, *mapDataIndex, pBlockIter->block.aSubBlock[0].offset); +#endif + + return TSDB_CODE_SUCCESS; } static int32_t initBlockIterator(STsdbReader* pReader, SDataBlockIter* pBlockIter, int32_t numOfBlocks) { @@ -867,10 +904,15 @@ static int32_t initBlockIterator(STsdbReader* pReader, SDataBlockIter* pBlockIte } sup.pDataBlockInfo[sup.numOfTables] = (SBlockOrderWrapper*)buf; + SBlock block = {0}; for (int32_t k = 0; k < num; ++k) { SBlockOrderWrapper wrapper = {0}; - wrapper.pBlock = (SBlock*)taosArrayGet(pTableScanInfo->pBlockList, k); + + int32_t* mapDataIndex = taosArrayGet(pTableScanInfo->pBlockList, k); + tMapDataGetItemByIdx(&pTableScanInfo->mapData, *mapDataIndex, &block, tGetBlock); + wrapper.uid = pTableScanInfo->uid; + wrapper.offset = block.aSubBlock[0].offset; sup.pDataBlockInfo[sup.numOfTables][k] = wrapper; cnt++; @@ -894,6 +936,7 @@ static int32_t initBlockIterator(STsdbReader* pReader, SDataBlockIter* pBlockIte pBlockIter->index = asc ? 0 : (numOfBlocks - 1); cleanupBlockOrderSupporter(&sup); + doSetCurrentBlock(pBlockIter); return TSDB_CODE_SUCCESS; } @@ -932,6 +975,8 @@ static int32_t initBlockIterator(STsdbReader* pReader, SDataBlockIter* pBlockIte taosMemoryFree(pTree); pBlockIter->index = asc ? 0 : (numOfBlocks - 1); + doSetCurrentBlock(pBlockIter); + return TSDB_CODE_SUCCESS; } @@ -944,6 +989,8 @@ static bool blockIteratorNext(SDataBlockIter* pBlockIter) { } pBlockIter->index += step; + doSetCurrentBlock(pBlockIter); + return true; } @@ -957,11 +1004,6 @@ static int32_t dataBlockPartiallyRequired(STimeWindow* pWindow, SVersionRange* p (pVerRange->maxVer < pBlock->maxVersion && pVerRange->maxVer >= pBlock->minVersion); } -static SFileDataBlockInfo* getCurrentBlockInfo(SDataBlockIter* pBlockIter) { - SFileDataBlockInfo* pFBlockInfo = taosArrayGet(pBlockIter->blockList, pBlockIter->index); - return pFBlockInfo; -} - static SBlock* getNeighborBlockOfSameTable(SFileDataBlockInfo* pFBlockInfo, STableBlockScanInfo* pTableBlockScanInfo, int32_t* nextIndex, int32_t order) { bool asc = ASCENDING_TRAVERSE(order); @@ -974,10 +1016,13 @@ static SBlock* getNeighborBlockOfSameTable(SFileDataBlockInfo* pFBlockInfo, STab } int32_t step = asc ? 1 : -1; - *nextIndex = pFBlockInfo->tbBlockIdx + step; - SBlock* pNext = taosArrayGet(pTableBlockScanInfo->pBlockList, *nextIndex); - return pNext; + + SBlock *pBlock = taosMemoryCalloc(1, sizeof(SBlock)); + int32_t* indexInMapdata = taosArrayGet(pTableBlockScanInfo->pBlockList, *nextIndex); + + tMapDataGetItemByIdx(&pTableBlockScanInfo->mapData, *indexInMapdata, pBlock, tGetBlock); + return pBlock; } static int32_t findFileBlockInfoIndex(SDataBlockIter* pBlockIter, SFileDataBlockInfo* pFBlockInfo) { @@ -1015,6 +1060,7 @@ static int32_t setFileBlockActiveInBlockIter(SDataBlockIter* pBlockIter, int32_t ASSERT(pBlockInfo->uid == fblock.uid && pBlockInfo->tbBlockIdx == fblock.tbBlockIdx); } + doSetCurrentBlock(pBlockIter); return TSDB_CODE_SUCCESS; } @@ -1117,6 +1163,7 @@ static bool fileBlockShouldLoad(STsdbReader* pReader, SFileDataBlockInfo* pFBloc bool overlapWithNeighbor = false; if (pNeighbor) { overlapWithNeighbor = overlapWithNeighborBlock(pBlock, pNeighbor, pReader->order); + taosMemoryFree(pNeighbor); } // has duplicated ts of different version in this block @@ -1142,11 +1189,13 @@ static int32_t buildDataBlockFromBuf(STsdbReader* pReader, STableBlockScanInfo* setComposedBlockFlag(pReader, true); - int64_t elapsedTime = taosGetTimestampUs() - st; - tsdbDebug("%p build data block from cache completed, elapsed time:%" PRId64 - " us, numOfRows:%d, numOfCols:%d, brange: %" PRId64 " - %" PRId64 " %s", - pReader, elapsedTime, pBlock->info.rows, (int32_t)blockDataGetNumOfCols(pBlock), pBlock->info.window.skey, - pBlock->info.window.ekey, pReader->idStr); + double elapsedTime = (taosGetTimestampUs() - st) / 1000.0; + tsdbDebug( + "%p build data block from cache completed, elapsed time:%.2f ms, numOfRows:%d, brange: %" PRId64 + " - %" PRId64 " %s", + pReader, elapsedTime, pBlock->info.rows, pBlock->info.window.skey, pBlock->info.window.ekey, pReader->idStr); + + pReader->cost.buildmemBlock += elapsedTime; return code; } @@ -1362,7 +1411,6 @@ static int32_t buildComposedDataBlockImpl(STsdbReader* pReader, STableBlockScanI SFileBlockDumpInfo* pDumpInfo = &pReader->status.fBlockDumpInfo; SBlockData* pBlockData = &pReader->status.fileBlockData; - SRowMerger merge = {0}; STSRow* pTSRow = NULL; int64_t key = pBlockData->aTSKEY[pDumpInfo->rowIndex]; @@ -1384,6 +1432,8 @@ static int32_t buildComposedDataBlockImpl(STsdbReader* pReader, STableBlockScanI // imem & mem are all empty, only file exist TSDBROW fRow = tsdbRowFromBlockData(pBlockData, pDumpInfo->rowIndex); + + SRowMerger merge = {0}; tRowMergerInit(&merge, &fRow, pReader->pSchema); doMergeRowsInFileBlocks(pBlockData, pBlockScanInfo, pReader, &merge); tRowMergerGetRow(&merge, &pTSRow); @@ -1408,9 +1458,7 @@ static int32_t buildComposedDataBlock(STsdbReader* pReader, STableBlockScanInfo* if (!isValidFileBlockRow(pBlockData, pDumpInfo, pBlockScanInfo, pReader)) { pDumpInfo->rowIndex += step; - SFileDataBlockInfo* pFBlock = getCurrentBlockInfo(&pReader->status.blockIter); - SBlock* pBlock = taosArrayGet(pBlockScanInfo->pBlockList, pFBlock->tbBlockIdx); - + SBlock* pBlock = getCurrentBlock(&pReader->status.blockIter); if (pDumpInfo->rowIndex >= pBlock->nRow || pDumpInfo->rowIndex < 0) { setBlockAllDumped(pDumpInfo, pBlock, pReader->order); break; @@ -1421,9 +1469,7 @@ static int32_t buildComposedDataBlock(STsdbReader* pReader, STableBlockScanInfo* } buildComposedDataBlockImpl(pReader, pBlockScanInfo); - - SFileDataBlockInfo* pFBlock = getCurrentBlockInfo(&pReader->status.blockIter); - SBlock* pBlock = taosArrayGet(pBlockScanInfo->pBlockList, pFBlock->tbBlockIdx); + SBlock* pBlock = getCurrentBlock(&pReader->status.blockIter); // currently loaded file data block is consumed if (pDumpInfo->rowIndex >= pBlock->nRow || pDumpInfo->rowIndex < 0) { @@ -1666,7 +1712,7 @@ static int32_t doBuildDataBlock(STsdbReader* pReader) { SFileDataBlockInfo* pFBlock = getCurrentBlockInfo(pBlockIter); STableBlockScanInfo* pScanInfo = taosHashGet(pStatus->pTableMap, &pFBlock->uid, sizeof(pFBlock->uid)); - SBlock* pBlock = taosArrayGet(pScanInfo->pBlockList, pFBlock->tbBlockIdx); + SBlock* pBlock = getCurrentBlock(pBlockIter); TSDBKEY key = getCurrentKeyInBuf(pBlockIter, pReader); if (fileBlockShouldLoad(pReader, pFBlock, pBlock, pScanInfo, key)) { @@ -1729,9 +1775,7 @@ static int32_t buildBlockFromBufferSequentially(STsdbReader* pReader) { // set the correct start position in case of the first/last file block, according to the query time window static void initBlockDumpInfo(STsdbReader* pReader, SDataBlockIter* pBlockIter) { - SFileDataBlockInfo* pFBlock = getCurrentBlockInfo(pBlockIter); - STableBlockScanInfo* pScanInfo = taosHashGet(pReader->status.pTableMap, &pFBlock->uid, sizeof(pFBlock->uid)); - SBlock* pBlock = taosArrayGet(pScanInfo->pBlockList, pFBlock->tbBlockIdx); + SBlock* pBlock = getCurrentBlock(pBlockIter); SReaderStatus* pStatus = &pReader->status; @@ -2102,6 +2146,8 @@ static int32_t checkForNeighborFileBlock(STsdbReader* pReader, STableBlockScanIn } bool overlap = overlapWithNeighborBlock(pBlock, pNeighborBlock, pReader->order); + taosMemoryFree(pNeighborBlock); + if (overlap) { // load next block SReaderStatus* pStatus = &pReader->status; SDataBlockIter* pBlockIter = &pStatus->blockIter; @@ -2152,7 +2198,7 @@ int32_t doMergeRowsInFileBlocks(SBlockData* pBlockData, STableBlockScanInfo* pSc CHECK_FILEBLOCK_STATE st; SFileDataBlockInfo* pFileBlockInfo = getCurrentBlockInfo(&pReader->status.blockIter); - SBlock* pCurrentBlock = taosArrayGet(pScanInfo->pBlockList, pFileBlockInfo->tbBlockIdx); + SBlock* pCurrentBlock = getCurrentBlock(&pReader->status.blockIter); checkForNeighborFileBlock(pReader, pScanInfo, pCurrentBlock, pFileBlockInfo, pMerger, key, &st); if (st == CHECK_FILEBLOCK_QUIT) { break; @@ -2461,7 +2507,7 @@ int32_t tsdbReaderOpen(SVnode* pVnode, SQueryTableDataCond* pCond, SArray* pTabl SDataBlockIter* pBlockIter = &pReader->status.blockIter; initFilesetIterator(&pReader->status.fileIter, pReader->pReadSnap->fs.aDFileSet, pReader->order, pReader->idStr); - resetDataBlockIterator(&pReader->status.blockIter, pReader->order); + resetDataBlockIterator(&pReader->status.blockIter, pReader->order, pReader->status.pTableMap); // no data in files, let's try buffer in memory if (pReader->status.fileIter.numOfFiles == 0) { @@ -2477,7 +2523,7 @@ int32_t tsdbReaderOpen(SVnode* pVnode, SQueryTableDataCond* pCond, SArray* pTabl SDataBlockIter* pBlockIter = &pPrevReader->status.blockIter; initFilesetIterator(&pPrevReader->status.fileIter, pPrevReader->pReadSnap->fs.aDFileSet, pPrevReader->order, pPrevReader->idStr); - resetDataBlockIterator(&pPrevReader->status.blockIter, pPrevReader->order); + resetDataBlockIterator(&pPrevReader->status.blockIter, pPrevReader->order, pReader->status.pTableMap); // no data in files, let's try buffer in memory if (pPrevReader->status.fileIter.numOfFiles == 0) { @@ -2519,6 +2565,8 @@ void tsdbReaderClose(STsdbReader* pReader) { taosMemoryFree(pSupInfo->buildBuf); cleanupDataBlockIterator(&pReader->status.blockIter); + + size_t numOfTables = taosHashGetSize(pReader->status.pTableMap); destroyBlockScanInfo(pReader->status.pTableMap); blockDataDestroy(pReader->pResBlock); @@ -2528,10 +2576,12 @@ void tsdbReaderClose(STsdbReader* pReader) { SIOCostSummary* pCost = &pReader->cost; - tsdbDebug("%p :io-cost summary: head-file read cnt:%" PRIu64 ", head-file time:%" PRIu64 " us, statis-info:%" PRId64 - " us, datablock:%" PRId64 " us, check data:%" PRId64 " us, %s", - pReader, pCost->headFileLoad, pCost->headFileLoadTime, pCost->smaLoadTime, pCost->blockLoadTime, - pCost->checkForNextTime, pReader->idStr); + tsdbDebug("%p :io-cost summary: head-file:%" PRIu64 ", head-file time:%.2f ms, SMA:%"PRId64" SMA-time:%.2f ms, " + "fileBlocks:%"PRId64", fileBlocks-time:%.2f ms, build in-memory-block-time:%.2f ms, STableBlockScanInfo " + "size:%.2f Kb %s", + pReader, pCost->headFileLoad, pCost->headFileLoadTime, pCost->smaData, pCost->smaLoadTime, + pCost->numOfBlocks, pCost->blockLoadTime, pCost->buildmemBlock, + numOfTables * sizeof(STableBlockScanInfo) /1000.0, pReader->idStr); taosMemoryFree(pReader->idStr); taosMemoryFree(pReader->pSchema); @@ -2543,7 +2593,6 @@ static bool doTsdbNextDataBlock(STsdbReader* pReader) { SSDataBlock* pBlock = pReader->pResBlock; blockDataCleanup(pBlock); - int64_t stime = taosGetTimestampUs(); SReaderStatus* pStatus = &pReader->status; if (pStatus->loadFromFile) { @@ -2639,9 +2688,8 @@ int32_t tsdbRetrieveDatablockSMA(STsdbReader* pReader, SColumnDataAgg*** pBlockS } SFileDataBlockInfo* pFBlock = getCurrentBlockInfo(&pReader->status.blockIter); - STableBlockScanInfo* pBlockScanInfo = taosHashGet(pReader->status.pTableMap, &pFBlock->uid, sizeof(pFBlock->uid)); - SBlock* pBlock = taosArrayGet(pBlockScanInfo->pBlockList, pFBlock->tbBlockIdx); + SBlock* pBlock = getCurrentBlock(&pReader->status.blockIter); int64_t stime = taosGetTimestampUs(); SBlockLoadSuppInfo* pSup = &pReader->suppInfo; @@ -2690,12 +2738,13 @@ int32_t tsdbRetrieveDatablockSMA(STsdbReader* pReader, SColumnDataAgg*** pBlockS } } - int64_t elapsed = taosGetTimestampUs() - stime; + double elapsed = (taosGetTimestampUs() - stime) / 1000.0; pReader->cost.smaLoadTime += elapsed; + pReader->cost.smaData += 1; *pBlockStatis = pSup->plist; - tsdbDebug("vgId:%d, succeed to load block SMA for uid %" PRIu64 ", elapsed time:%" PRId64 "us, %s", 0, pFBlock->uid, + tsdbDebug("vgId:%d, succeed to load block SMA for uid %" PRIu64 ", elapsed time:%.2f ms, %s", 0, pFBlock->uid, elapsed, pReader->idStr); return code; @@ -2764,7 +2813,7 @@ int32_t tsdbReaderReset(STsdbReader* pReader, SQueryTableDataCond* pCond) { tsdbDataFReaderClose(&pReader->pFileReader); initFilesetIterator(&pReader->status.fileIter, pReader->pReadSnap->fs.aDFileSet, pReader->order, pReader->idStr); - resetDataBlockIterator(&pReader->status.blockIter, pReader->order); + resetDataBlockIterator(&pReader->status.blockIter, pReader->order, pReader->status.pTableMap); resetDataBlockScanInfo(pReader->status.pTableMap); int32_t code = 0; diff --git a/source/dnode/vnode/src/tsdb/tsdbUtil.c b/source/dnode/vnode/src/tsdb/tsdbUtil.c index 872357fc93bd4a1061b388b0961b888a954d6f00..805e49a705ebaadc4dd8447f2beb31174cad6608 100644 --- a/source/dnode/vnode/src/tsdb/tsdbUtil.c +++ b/source/dnode/vnode/src/tsdb/tsdbUtil.c @@ -24,6 +24,8 @@ void tMapDataReset(SMapData *pMapData) { void tMapDataClear(SMapData *pMapData) { tFree((uint8_t *)pMapData->aOffset); tFree(pMapData->pData); + pMapData->pData = NULL; + pMapData->aOffset = NULL; } int32_t tMapDataPutItem(SMapData *pMapData, void *pItem, int32_t (*tPutItemFn)(uint8_t *, void *)) { diff --git a/source/dnode/vnode/src/vnd/vnodeCfg.c b/source/dnode/vnode/src/vnd/vnodeCfg.c index eac1fd1a74dc9f83a15492c4e686b08d572734aa..e38fe9876b31be7d171a748c29a18ea5fc31c770 100644 --- a/source/dnode/vnode/src/vnd/vnodeCfg.c +++ b/source/dnode/vnode/src/vnd/vnodeCfg.c @@ -40,8 +40,8 @@ const SVnodeCfg vnodeCfgDefault = {.vgId = -1, .vgId = -1, .fsyncPeriod = 0, .retentionPeriod = -1, - .rollPeriod = -1, - .segSize = -1, + .rollPeriod = 0, + .segSize = 0, .retentionSize = -1, .level = TAOS_WAL_WRITE, }, diff --git a/source/dnode/vnode/src/vnd/vnodeSync.c b/source/dnode/vnode/src/vnd/vnodeSync.c index a0e2354f5132f52f6bce9341be78e49dd1a8ebbf..2b760efba0830f0d169bc005051fc6725bf4ff8f 100644 --- a/source/dnode/vnode/src/vnd/vnodeSync.c +++ b/source/dnode/vnode/src/vnd/vnodeSync.c @@ -206,13 +206,13 @@ static void inline vnodeProposeBatchMsg(SVnode *pVnode, SRpcMsg **pMsgArr, bool } void vnodeProposeWriteMsg(SQueueInfo *pInfo, STaosQall *qall, int32_t numOfMsgs) { - SVnode *pVnode = pInfo->ahandle; - int32_t vgId = pVnode->config.vgId; - int32_t code = 0; - SRpcMsg *pMsg = NULL; - int32_t arrayPos = 0; - SRpcMsg **pMsgArr = taosMemoryCalloc(numOfMsgs, sizeof(SRpcMsg*)); - bool *pIsWeakArr = taosMemoryCalloc(numOfMsgs, sizeof(bool)); + SVnode *pVnode = pInfo->ahandle; + int32_t vgId = pVnode->config.vgId; + int32_t code = 0; + SRpcMsg *pMsg = NULL; + int32_t arrayPos = 0; + SRpcMsg **pMsgArr = taosMemoryCalloc(numOfMsgs, sizeof(SRpcMsg *)); + bool *pIsWeakArr = taosMemoryCalloc(numOfMsgs, sizeof(bool)); vTrace("vgId:%d, get %d msgs from vnode-write queue", vgId, numOfMsgs); for (int32_t msg = 0; msg < numOfMsgs; msg++) { @@ -506,7 +506,7 @@ static void vnodeSyncCommitMsg(SSyncFSM *pFsm, const SRpcMsg *pMsg, SFsmCbMeta c syncGetVgId(pVnode->sync), pFsm, cbMeta.index, cbMeta.isWeak, cbMeta.code, cbMeta.state, syncUtilState2String(cbMeta.state), pMsg->msgType, TMSG_INFO(pMsg->msgType)); - if (cbMeta.code == 0) { + if (cbMeta.code == 0 && cbMeta.isWeak == 0) { SRpcMsg rpcMsg = {.msgType = pMsg->msgType, .contLen = pMsg->contLen}; rpcMsg.pCont = rpcMallocCont(rpcMsg.contLen); memcpy(rpcMsg.pCont, pMsg->pCont, pMsg->contLen); @@ -529,6 +529,23 @@ static void vnodeSyncPreCommitMsg(SSyncFSM *pFsm, const SRpcMsg *pMsg, SFsmCbMet vTrace("vgId:%d, pre-commit-cb is excuted, fsm:%p, index:%" PRId64 ", isWeak:%d, code:%d, state:%d %s, msgtype:%d %s", syncGetVgId(pVnode->sync), pFsm, cbMeta.index, cbMeta.isWeak, cbMeta.code, cbMeta.state, syncUtilState2String(cbMeta.state), pMsg->msgType, TMSG_INFO(pMsg->msgType)); + + if (cbMeta.code == 0 && cbMeta.isWeak == 1) { + SRpcMsg rpcMsg = {.msgType = pMsg->msgType, .contLen = pMsg->contLen}; + rpcMsg.pCont = rpcMallocCont(rpcMsg.contLen); + memcpy(rpcMsg.pCont, pMsg->pCont, pMsg->contLen); + syncGetAndDelRespRpc(pVnode->sync, cbMeta.seqNum, &rpcMsg.info); + rpcMsg.info.conn.applyIndex = cbMeta.index; + rpcMsg.info.conn.applyTerm = cbMeta.term; + tmsgPutToQueue(&pVnode->msgCb, APPLY_QUEUE, &rpcMsg); + } else { + SRpcMsg rsp = {.code = cbMeta.code, .info = pMsg->info}; + vError("vgId:%d, sync pre-commit error, msgtype:%d,%s, error:0x%X, errmsg:%s", syncGetVgId(pVnode->sync), + pMsg->msgType, TMSG_INFO(pMsg->msgType), cbMeta.code, tstrerror(cbMeta.code)); + if (rsp.info.handle != NULL) { + tmsgSendRsp(&rsp); + } + } } static void vnodeSyncRollBackMsg(SSyncFSM *pFsm, const SRpcMsg *pMsg, SFsmCbMeta cbMeta) { diff --git a/source/libs/executor/inc/executorimpl.h b/source/libs/executor/inc/executorimpl.h index f1a1011ff0abafebc42b4772dfff59e6519e5cac..6c0f4b2c122739af4415d4dffa4f8108bf31e3b4 100644 --- a/source/libs/executor/inc/executorimpl.h +++ b/source/libs/executor/inc/executorimpl.h @@ -1003,6 +1003,7 @@ int32_t updateSessionWindowInfo(SResultWindowInfo* pWinInfo, TSKEY* pStartTs, bool functionNeedToExecute(SqlFunctionCtx* pCtx); bool isCloseWindow(STimeWindow* pWin, STimeWindowAggSupp* pSup); void appendOneRow(SSDataBlock* pBlock, TSKEY* pStartTs, TSKEY* pEndTs, uint64_t* pUid); +void printDataBlock(SSDataBlock* pBlock, const char* flag); int32_t finalizeResultRowIntoResultDataBlock(SDiskbasedBuf* pBuf, SResultRowPosition* resultRowPosition, SqlFunctionCtx* pCtx, SExprInfo* pExprInfo, int32_t numOfExprs, const int32_t* rowCellOffset, diff --git a/source/libs/executor/src/dataDeleter.c b/source/libs/executor/src/dataDeleter.c index 3c56abbd15395d7c1a1f8942e8ea9587a4e1a2ce..a0d4d64bff1d709e820c1a1616254fb9a206fc90 100644 --- a/source/libs/executor/src/dataDeleter.c +++ b/source/libs/executor/src/dataDeleter.c @@ -182,7 +182,8 @@ static int32_t getDataBlock(SDataSinkHandle* pHandle, SOutputData* pOutput) { return TSDB_CODE_SUCCESS; } SDataCacheEntry* pEntry = (SDataCacheEntry*)(pDeleter->nextOutput.pData); - memcpy(pOutput->pData, pEntry->data, pEntry->dataLen); + memcpy(pOutput->pData, pEntry->data, pEntry->dataLen); + pDeleter->pParam->pUidList = NULL; pOutput->numOfRows = pEntry->numOfRows; pOutput->numOfCols = pEntry->numOfCols; pOutput->compressed = pEntry->compressed; @@ -205,6 +206,8 @@ static int32_t destroyDataSinker(SDataSinkHandle* pHandle) { SDataDeleterHandle* pDeleter = (SDataDeleterHandle*)pHandle; atomic_sub_fetch_64(&gDataSinkStat.cachedSize, pDeleter->cachedSize); taosMemoryFreeClear(pDeleter->nextOutput.pData); + taosArrayDestroy(pDeleter->pParam->pUidList); + taosMemoryFree(pDeleter->pParam); while (!taosQueueEmpty(pDeleter->pDataBlocks)) { SDataDeleterBuf* pBuf = NULL; taosReadQitem(pDeleter->pDataBlocks, (void**)&pBuf); diff --git a/source/libs/executor/src/executorimpl.c b/source/libs/executor/src/executorimpl.c index 8be54661828280eacb6fdfe2eda58858db497eee..3f2f528a2d6211a227b931d7a13bebe4e41261eb 100644 --- a/source/libs/executor/src/executorimpl.c +++ b/source/libs/executor/src/executorimpl.c @@ -666,6 +666,11 @@ int32_t projectApplyFunctions(SExprInfo* pExpr, SSDataBlock* pResult, SSDataBloc pfCtx->pTsOutput = (SColumnInfoData*)pCtx[*outputColIndex].pOutput; } + // link pDstBlock to set selectivity value + if (pfCtx->subsidiaries.num > 0) { + pfCtx->pDstBlock = pResult; + } + numOfRows = pfCtx->fpSet.process(pfCtx); } else if (fmIsAggFunc(pfCtx->functionId)) { // _group_key function for "partition by tbname" + csum(col_name) query diff --git a/source/libs/executor/src/timewindowoperator.c b/source/libs/executor/src/timewindowoperator.c index 1e001a29a09ccf08641ea007f498c1fa44b96ed9..c2cf19167a54abde07b2b8281138889f8d7c87a0 100644 --- a/source/libs/executor/src/timewindowoperator.c +++ b/source/libs/executor/src/timewindowoperator.c @@ -1616,6 +1616,7 @@ static SSDataBlock* doStreamIntervalAgg(SOperatorInfo* pOperator) { pInfo->twAggSup.maxTs = TMAX(pInfo->twAggSup.maxTs, pBlock->info.window.ekey); hashIntervalAgg(pOperator, &pInfo->binfo.resultRowInfo, pBlock, MAIN_SCAN, pUpdated); } + pOperator->status = OP_RES_TO_RETURN; closeIntervalWindow(pInfo->aggSup.pResultRowHashTable, &pInfo->twAggSup, &pInfo->interval, NULL, pUpdated, pInfo->pRecycledPages, pInfo->aggSup.pResultBuf); @@ -1628,6 +1629,7 @@ static SSDataBlock* doStreamIntervalAgg(SOperatorInfo* pOperator) { if (pInfo->pDelRes->info.rows > 0) { return pInfo->pDelRes; } + doBuildResultDatablock(pOperator, &pInfo->binfo, &pInfo->groupResInfo, pInfo->aggSup.pResultBuf); printDataBlock(pInfo->binfo.pRes, "single interval"); return pInfo->binfo.pRes->info.rows == 0 ? NULL : pInfo->binfo.pRes; @@ -2769,7 +2771,7 @@ static SSDataBlock* doStreamFinalIntervalAgg(SOperatorInfo* pOperator) { SExprSupp* pSup = &pOperator->exprSupp; - qDebug("interval status %d %s", pOperator->status, IS_FINAL_OP(pInfo) ? "interval Final" : "interval Semi"); + qDebug("interval status %d %s", pOperator->status, IS_FINAL_OP(pInfo) ? "interval final" : "interval semi"); if (pOperator->status == OP_EXEC_DONE) { return NULL; @@ -2778,7 +2780,7 @@ static SSDataBlock* doStreamFinalIntervalAgg(SOperatorInfo* pOperator) { if (pInfo->pPullDataRes->info.rows != 0) { // process the rest of the data ASSERT(IS_FINAL_OP(pInfo)); - printDataBlock(pInfo->pPullDataRes, IS_FINAL_OP(pInfo) ? "interval Final" : "interval Semi"); + printDataBlock(pInfo->pPullDataRes, IS_FINAL_OP(pInfo) ? "interval final" : "interval semi"); return pInfo->pPullDataRes; } @@ -2793,20 +2795,20 @@ static SSDataBlock* doStreamFinalIntervalAgg(SOperatorInfo* pOperator) { } return NULL; } - printDataBlock(pInfo->binfo.pRes, IS_FINAL_OP(pInfo) ? "interval Final" : "interval Semi"); + printDataBlock(pInfo->binfo.pRes, IS_FINAL_OP(pInfo) ? "interval final" : "interval semi"); return pInfo->binfo.pRes; } else { if (!IS_FINAL_OP(pInfo)) { doBuildResultDatablock(pOperator, &pInfo->binfo, &pInfo->groupResInfo, pInfo->aggSup.pResultBuf); if (pInfo->binfo.pRes->info.rows != 0) { - printDataBlock(pInfo->binfo.pRes, IS_FINAL_OP(pInfo) ? "interval Final" : "interval Semi"); + printDataBlock(pInfo->binfo.pRes, IS_FINAL_OP(pInfo) ? "interval final" : "interval semi"); return pInfo->binfo.pRes; } } if (pInfo->pUpdateRes->info.rows != 0 && pInfo->returnUpdate) { pInfo->returnUpdate = false; ASSERT(!IS_FINAL_OP(pInfo)); - printDataBlock(pInfo->pUpdateRes, IS_FINAL_OP(pInfo) ? "interval Final" : "interval Semi"); + printDataBlock(pInfo->pUpdateRes, IS_FINAL_OP(pInfo) ? "interval final" : "interval semi"); // process the rest of the data return pInfo->pUpdateRes; } @@ -2814,13 +2816,13 @@ static SSDataBlock* doStreamFinalIntervalAgg(SOperatorInfo* pOperator) { // if (pInfo->pPullDataRes->info.rows != 0) { // // process the rest of the data // ASSERT(IS_FINAL_OP(pInfo)); - // printDataBlock(pInfo->pPullDataRes, IS_FINAL_OP(pInfo) ? "interval Final" : "interval Semi"); + // printDataBlock(pInfo->pPullDataRes, IS_FINAL_OP(pInfo) ? "interval final" : "interval semi"); // return pInfo->pPullDataRes; // } doBuildDeleteResult(pInfo->pDelWins, &pInfo->delIndex, pInfo->pDelRes); if (pInfo->pDelRes->info.rows != 0) { // process the rest of the data - printDataBlock(pInfo->pDelRes, IS_FINAL_OP(pInfo) ? "interval Final" : "interval Semi"); + printDataBlock(pInfo->pDelRes, IS_FINAL_OP(pInfo) ? "interval final" : "interval semi"); return pInfo->pDelRes; } } @@ -2831,10 +2833,10 @@ static SSDataBlock* doStreamFinalIntervalAgg(SOperatorInfo* pOperator) { clearSpecialDataBlock(pInfo->pUpdateRes); removeDeleteResults(pUpdated, pInfo->pDelWins); pOperator->status = OP_RES_TO_RETURN; - qDebug("%s return data", IS_FINAL_OP(pInfo) ? "interval Final" : "interval Semi"); + qDebug("%s return data", IS_FINAL_OP(pInfo) ? "interval final" : "interval semi"); break; } - printDataBlock(pBlock, IS_FINAL_OP(pInfo) ? "interval Final recv" : "interval Semi recv"); + printDataBlock(pBlock, IS_FINAL_OP(pInfo) ? "interval final recv" : "interval semi recv"); maxTs = TMAX(maxTs, pBlock->info.window.ekey); if (pBlock->info.type == STREAM_NORMAL || pBlock->info.type == STREAM_PULL_DATA || @@ -2934,20 +2936,20 @@ static SSDataBlock* doStreamFinalIntervalAgg(SOperatorInfo* pOperator) { if (pInfo->pPullDataRes->info.rows != 0) { // process the rest of the data ASSERT(IS_FINAL_OP(pInfo)); - printDataBlock(pInfo->pPullDataRes, IS_FINAL_OP(pInfo) ? "interval Final" : "interval Semi"); + printDataBlock(pInfo->pPullDataRes, IS_FINAL_OP(pInfo) ? "interval final" : "interval semi"); return pInfo->pPullDataRes; } doBuildResultDatablock(pOperator, &pInfo->binfo, &pInfo->groupResInfo, pInfo->aggSup.pResultBuf); if (pInfo->binfo.pRes->info.rows != 0) { - printDataBlock(pInfo->binfo.pRes, IS_FINAL_OP(pInfo) ? "interval Final" : "interval Semi"); + printDataBlock(pInfo->binfo.pRes, IS_FINAL_OP(pInfo) ? "interval final" : "interval semi"); return pInfo->binfo.pRes; } if (pInfo->pUpdateRes->info.rows != 0 && pInfo->returnUpdate) { pInfo->returnUpdate = false; ASSERT(!IS_FINAL_OP(pInfo)); - printDataBlock(pInfo->pUpdateRes, IS_FINAL_OP(pInfo) ? "interval Final" : "interval Semi"); + printDataBlock(pInfo->pUpdateRes, IS_FINAL_OP(pInfo) ? "interval final" : "interval semi"); // process the rest of the data return pInfo->pUpdateRes; } @@ -2955,7 +2957,7 @@ static SSDataBlock* doStreamFinalIntervalAgg(SOperatorInfo* pOperator) { doBuildDeleteResult(pInfo->pDelWins, &pInfo->delIndex, pInfo->pDelRes); if (pInfo->pDelRes->info.rows != 0) { // process the rest of the data - printDataBlock(pInfo->pDelRes, IS_FINAL_OP(pInfo) ? "interval Final" : "interval Semi"); + printDataBlock(pInfo->pDelRes, IS_FINAL_OP(pInfo) ? "interval final" : "interval semi"); return pInfo->pDelRes; } // ASSERT(false); @@ -3815,14 +3817,14 @@ static SSDataBlock* doStreamSessionAgg(SOperatorInfo* pOperator) { } else if (pOperator->status == OP_RES_TO_RETURN) { doBuildDeleteDataBlock(pInfo->pStDeleted, pInfo->pDelRes, &pInfo->pDelIterator); if (pInfo->pDelRes->info.rows > 0) { - printDataBlock(pInfo->pDelRes, IS_FINAL_OP(pInfo) ? "Final Session" : "Single Session"); + printDataBlock(pInfo->pDelRes, IS_FINAL_OP(pInfo) ? "final session" : "single session"); return pInfo->pDelRes; } doBuildResultDatablock(pOperator, pBInfo, &pInfo->groupResInfo, pInfo->streamAggSup.pResultBuf); if (pBInfo->pRes->info.rows == 0 || !hasDataInGroupInfo(&pInfo->groupResInfo)) { doSetOperatorCompleted(pOperator); } - printDataBlock(pBInfo->pRes, IS_FINAL_OP(pInfo) ? "Final Session" : "Single Session"); + printDataBlock(pBInfo->pRes, IS_FINAL_OP(pInfo) ? "final session" : "single session"); return pBInfo->pRes->info.rows == 0 ? NULL : pBInfo->pRes; } @@ -3835,7 +3837,7 @@ static SSDataBlock* doStreamSessionAgg(SOperatorInfo* pOperator) { if (pBlock == NULL) { break; } - printDataBlock(pBlock, IS_FINAL_OP(pInfo) ? "Final Session Recv" : "Single Session Recv"); + printDataBlock(pBlock, IS_FINAL_OP(pInfo) ? "final session recv" : "single session recv"); if (pBlock->info.type == STREAM_CLEAR) { SArray* pWins = taosArrayInit(16, sizeof(SResultWindowInfo)); @@ -3912,11 +3914,11 @@ static SSDataBlock* doStreamSessionAgg(SOperatorInfo* pOperator) { blockDataEnsureCapacity(pInfo->binfo.pRes, pOperator->resultInfo.capacity); doBuildDeleteDataBlock(pInfo->pStDeleted, pInfo->pDelRes, &pInfo->pDelIterator); if (pInfo->pDelRes->info.rows > 0) { - printDataBlock(pInfo->pDelRes, IS_FINAL_OP(pInfo) ? "Final Session" : "Single Session"); + printDataBlock(pInfo->pDelRes, IS_FINAL_OP(pInfo) ? "final session" : "single session"); return pInfo->pDelRes; } doBuildResultDatablock(pOperator, &pInfo->binfo, &pInfo->groupResInfo, pInfo->streamAggSup.pResultBuf); - printDataBlock(pBInfo->pRes, IS_FINAL_OP(pInfo) ? "Final Session" : "Single Session"); + printDataBlock(pBInfo->pRes, IS_FINAL_OP(pInfo) ? "final session" : "single session"); return pBInfo->pRes->info.rows == 0 ? NULL : pBInfo->pRes; } @@ -3955,21 +3957,21 @@ static SSDataBlock* doStreamSessionSemiAgg(SOperatorInfo* pOperator) { } else if (pOperator->status == OP_RES_TO_RETURN) { doBuildResultDatablock(pOperator, pBInfo, &pInfo->groupResInfo, pInfo->streamAggSup.pResultBuf); if (pBInfo->pRes->info.rows > 0) { - printDataBlock(pBInfo->pRes, "Semi Session"); + printDataBlock(pBInfo->pRes, "sems session"); return pBInfo->pRes; } // doBuildDeleteDataBlock(pInfo->pStDeleted, pInfo->pDelRes, &pInfo->pDelIterator); if (pInfo->pDelRes->info.rows > 0 && !pInfo->returnDelete) { pInfo->returnDelete = true; - printDataBlock(pInfo->pDelRes, "Semi Session"); + printDataBlock(pInfo->pDelRes, "sems session"); return pInfo->pDelRes; } if (pInfo->pUpdateRes->info.rows > 0) { // process the rest of the data pOperator->status = OP_OPENED; - printDataBlock(pInfo->pUpdateRes, "Semi Session"); + printDataBlock(pInfo->pUpdateRes, "sems session"); return pInfo->pUpdateRes; } // semi interval operator clear disk buffer @@ -4033,21 +4035,21 @@ static SSDataBlock* doStreamSessionSemiAgg(SOperatorInfo* pOperator) { doBuildResultDatablock(pOperator, pBInfo, &pInfo->groupResInfo, pInfo->streamAggSup.pResultBuf); if (pBInfo->pRes->info.rows > 0) { - printDataBlock(pBInfo->pRes, "Semi Session"); + printDataBlock(pBInfo->pRes, "sems session"); return pBInfo->pRes; } // doBuildDeleteDataBlock(pInfo->pStDeleted, pInfo->pDelRes, &pInfo->pDelIterator); if (pInfo->pDelRes->info.rows > 0 && !pInfo->returnDelete) { pInfo->returnDelete = true; - printDataBlock(pInfo->pDelRes, "Semi Session"); + printDataBlock(pInfo->pDelRes, "sems session"); return pInfo->pDelRes; } if (pInfo->pUpdateRes->info.rows > 0) { // process the rest of the data pOperator->status = OP_OPENED; - printDataBlock(pInfo->pUpdateRes, "Semi Session"); + printDataBlock(pInfo->pUpdateRes, "sems session"); return pInfo->pUpdateRes; } diff --git a/source/libs/function/src/builtins.c b/source/libs/function/src/builtins.c index 9f4fd28416b6be12345611b542b65ad2da3dcae9..01a5e7997ec041782137e89dc73c2e3da9c68b4e 100644 --- a/source/libs/function/src/builtins.c +++ b/source/libs/function/src/builtins.c @@ -2231,7 +2231,7 @@ const SBuiltinFuncDefinition funcMgtBuiltins[] = { { .name = "derivative", .type = FUNCTION_TYPE_DERIVATIVE, - .classification = FUNC_MGT_INDEFINITE_ROWS_FUNC | FUNC_MGT_TIMELINE_FUNC | FUNC_MGT_IMPLICIT_TS_FUNC, + .classification = FUNC_MGT_INDEFINITE_ROWS_FUNC | FUNC_MGT_SELECT_FUNC | FUNC_MGT_TIMELINE_FUNC | FUNC_MGT_IMPLICIT_TS_FUNC, .translateFunc = translateDerivative, .getEnvFunc = getDerivativeFuncEnv, .initFunc = derivativeFuncSetup, @@ -2436,7 +2436,7 @@ const SBuiltinFuncDefinition funcMgtBuiltins[] = { { .name = "diff", .type = FUNCTION_TYPE_DIFF, - .classification = FUNC_MGT_INDEFINITE_ROWS_FUNC | FUNC_MGT_TIMELINE_FUNC | FUNC_MGT_FORBID_STREAM_FUNC, + .classification = FUNC_MGT_INDEFINITE_ROWS_FUNC | FUNC_MGT_SELECT_FUNC | FUNC_MGT_TIMELINE_FUNC | FUNC_MGT_FORBID_STREAM_FUNC, .translateFunc = translateDiff, .getEnvFunc = getDiffFuncEnv, .initFunc = diffFunctionSetup, diff --git a/source/libs/function/src/builtinsimpl.c b/source/libs/function/src/builtinsimpl.c index 92f9bdd1b2fa12b0a7a5627087df9729bf32629c..536dd4299ac14a93ee5918dd042203c4114c32d2 100644 --- a/source/libs/function/src/builtinsimpl.c +++ b/source/libs/function/src/builtinsimpl.c @@ -1624,6 +1624,10 @@ int32_t minmaxFunctionFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock) { } void setNullSelectivityValue(SqlFunctionCtx* pCtx, SSDataBlock* pBlock, int32_t rowIndex) { + if (pCtx->subsidiaries.num <= 0) { + return; + } + for (int32_t j = 0; j < pCtx->subsidiaries.num; ++j) { SqlFunctionCtx* pc = pCtx->subsidiaries.pCtx[j]; int32_t dstSlotId = pc->pExpr->base.resSchema.slotId; @@ -1655,8 +1659,6 @@ void setSelectivityValue(SqlFunctionCtx* pCtx, SSDataBlock* pBlock, const STuple SFunctParam* pFuncParam = &pc->pExpr->base.pParam[0]; int32_t dstSlotId = pc->pExpr->base.resSchema.slotId; - int32_t ps = 0; - SColumnInfoData* pDstCol = taosArrayGet(pBlock->pDataBlock, dstSlotId); ASSERT(pc->pExpr->base.resSchema.bytes == pDstCol->info.bytes); if (nullList[j]) { @@ -1678,6 +1680,39 @@ void releaseSource(STuplePos* pPos) { // Todo(liuyao) relase row } +// This function append the selectivity to subsidiaries function context directly, without fetching data +// from intermediate disk based buf page +void appendSelectivityValue(SqlFunctionCtx* pCtx, int32_t rowIndex, int32_t pos) { + if (pCtx->subsidiaries.num <= 0) { + return; + } + + for (int32_t j = 0; j < pCtx->subsidiaries.num; ++j) { + SqlFunctionCtx* pc = pCtx->subsidiaries.pCtx[j]; + + // get data from source col + SFunctParam* pFuncParam = &pc->pExpr->base.pParam[0]; + int32_t srcSlotId = pFuncParam->pCol->slotId; + + SColumnInfoData* pSrcCol = taosArrayGet(pCtx->pSrcBlock->pDataBlock, srcSlotId); + + char* pData = colDataGetData(pSrcCol, rowIndex); + + // append to dest col + int32_t dstSlotId = pc->pExpr->base.resSchema.slotId; + + SColumnInfoData* pDstCol = taosArrayGet(pCtx->pDstBlock->pDataBlock, dstSlotId); + ASSERT(pc->pExpr->base.resSchema.bytes == pDstCol->info.bytes); + + if (colDataIsNull_s(pSrcCol, rowIndex) == true) { + colDataAppendNULL(pDstCol, pos); + } else { + colDataAppend(pDstCol, pos, pData, false); + } + } + +} + void replaceTupleData(STuplePos* pDestPos, STuplePos* pSourcePos) { releaseSource(pDestPos); *pDestPos = *pSourcePos; @@ -3155,6 +3190,7 @@ static void doHandleDiff(SDiffInfo* pDiffInfo, int32_t type, const char* pv, SCo colDataAppendInt64(pOutput, pos, &delta); } pDiffInfo->prev.i64 = v; + break; } case TSDB_DATA_TYPE_BOOL: @@ -3248,6 +3284,10 @@ int32_t diffFunction(SqlFunctionCtx* pCtx) { if (pDiffInfo->hasPrev) { doHandleDiff(pDiffInfo, pInputCol->info.type, pv, pOutput, pos, pCtx->order); + // handle selectivity + if (pCtx->subsidiaries.num > 0) { + appendSelectivityValue(pCtx, i, pos); + } numOfElems++; } else { @@ -3274,6 +3314,10 @@ int32_t diffFunction(SqlFunctionCtx* pCtx) { // there is a row of previous data block to be handled in the first place. if (pDiffInfo->hasPrev) { doHandleDiff(pDiffInfo, pInputCol->info.type, pv, pOutput, pos, pCtx->order); + // handle selectivity + if (pCtx->subsidiaries.num > 0) { + appendSelectivityValue(pCtx, i, pos); + } numOfElems++; } else { @@ -5724,6 +5768,12 @@ int32_t derivativeFunction(SqlFunctionCtx* pCtx) { if (pTsOutput != NULL) { colDataAppendInt64(pTsOutput, pos, &tsList[i]); } + + // handle selectivity + if (pCtx->subsidiaries.num > 0) { + appendSelectivityValue(pCtx, i, pos); + } + numOfElems++; } } @@ -5756,6 +5806,12 @@ int32_t derivativeFunction(SqlFunctionCtx* pCtx) { if (pTsOutput != NULL) { colDataAppendInt64(pTsOutput, pos, &pDerivInfo->prevTs); } + + // handle selectivity + if (pCtx->subsidiaries.num > 0) { + appendSelectivityValue(pCtx, i, pos); + } + numOfElems++; } } diff --git a/source/libs/parser/test/parSelectTest.cpp b/source/libs/parser/test/parSelectTest.cpp index 849ba14d119ebd6ebb1cc1efab99440c42ef298f..951ca5e40dbac005d67f028beff19bdd16228ab9 100644 --- a/source/libs/parser/test/parSelectTest.cpp +++ b/source/libs/parser/test/parSelectTest.cpp @@ -144,9 +144,9 @@ TEST_F(ParserSelectTest, IndefiniteRowsFunc) { TEST_F(ParserSelectTest, IndefiniteRowsFuncSemanticCheck) { useDb("root", "test"); - run("SELECT DIFF(c1), c2 FROM t1", TSDB_CODE_PAR_NOT_SINGLE_GROUP); + run("SELECT DIFF(c1), c2 FROM t1"); - run("SELECT DIFF(c1), tbname FROM t1", TSDB_CODE_PAR_NOT_SINGLE_GROUP); + run("SELECT DIFF(c1), tbname FROM t1"); run("SELECT DIFF(c1), count(*) FROM t1", TSDB_CODE_PAR_NOT_ALLOWED_FUNC); diff --git a/source/libs/qworker/src/qworker.c b/source/libs/qworker/src/qworker.c index d77e42388b454d6a6d6b3c39ee68d8b3963a7dda..e09887e6515cf1b75e97e0503eac3c846aac4c72 100644 --- a/source/libs/qworker/src/qworker.c +++ b/source/libs/qworker/src/qworker.c @@ -283,6 +283,8 @@ int32_t qwGetDeleteResFromSink(QW_FPARAMS_DEF, SQWTaskCtx *ctx, SDeleteRes *pRes pRes->skey = pDelRes->skey; pRes->ekey = pDelRes->ekey; pRes->affectedRows = pDelRes->affectedRows; + + taosMemoryFree(output.pData); return TSDB_CODE_SUCCESS; } diff --git a/source/libs/sync/inc/syncInt.h b/source/libs/sync/inc/syncInt.h index 64f66e390aafde8efdfcf215ec5387c3cfbd631f..b802d94bea2b65deb73594dcebfc8a393cb437d0 100644 --- a/source/libs/sync/inc/syncInt.h +++ b/source/libs/sync/inc/syncInt.h @@ -238,6 +238,7 @@ int32_t syncNodeGetPreIndexTerm(SSyncNode* pSyncNode, SyncIndex index, SyncInd bool syncNodeIsOptimizedOneReplica(SSyncNode* ths, SRpcMsg* pMsg); int32_t syncNodeCommit(SSyncNode* ths, SyncIndex beginIndex, SyncIndex endIndex, uint64_t flag); +int32_t syncNodePreCommit(SSyncNode* ths, SSyncRaftEntry* pEntry, int32_t code); int32_t syncNodeUpdateNewConfigIndex(SSyncNode* ths, SSyncCfg* pNewCfg); diff --git a/source/libs/sync/src/syncAppendEntries.c b/source/libs/sync/src/syncAppendEntries.c index 4295abeaa1a9eb3b4f0ac448a2c286da13f8a778..7d0f53640cc0a6e441c31dd3c52166c824dae4d6 100644 --- a/source/libs/sync/src/syncAppendEntries.c +++ b/source/libs/sync/src/syncAppendEntries.c @@ -244,22 +244,7 @@ int32_t syncNodeOnAppendEntriesCb(SSyncNode* ths, SyncAppendEntries* pMsg) { ths->pLogStore->appendEntry(ths->pLogStore, pAppendEntry); // pre commit - SRpcMsg rpcMsg; - syncEntry2OriginalRpc(pAppendEntry, &rpcMsg); - if (ths->pFsm != NULL) { - // if (ths->pFsm->FpPreCommitCb != NULL && pAppendEntry->originalRpcType != TDMT_SYNC_NOOP) { - if (ths->pFsm->FpPreCommitCb != NULL && syncUtilUserPreCommit(pAppendEntry->originalRpcType)) { - SFsmCbMeta cbMeta = {0}; - cbMeta.index = pAppendEntry->index; - cbMeta.lastConfigIndex = syncNodeGetSnapshotConfigIndex(ths, cbMeta.index); - cbMeta.isWeak = pAppendEntry->isWeak; - cbMeta.code = 2; - cbMeta.state = ths->state; - cbMeta.seqNum = pAppendEntry->seqNum; - ths->pFsm->FpPreCommitCb(ths->pFsm, &rpcMsg, cbMeta); - } - } - rpcFreeCont(rpcMsg.pCont); + syncNodePreCommit(ths, pAppendEntry, 0); } // free memory @@ -280,22 +265,7 @@ int32_t syncNodeOnAppendEntriesCb(SSyncNode* ths, SyncAppendEntries* pMsg) { ths->pLogStore->appendEntry(ths->pLogStore, pAppendEntry); // pre commit - SRpcMsg rpcMsg; - syncEntry2OriginalRpc(pAppendEntry, &rpcMsg); - if (ths->pFsm != NULL) { - // if (ths->pFsm->FpPreCommitCb != NULL && pAppendEntry->originalRpcType != TDMT_SYNC_NOOP) { - if (ths->pFsm->FpPreCommitCb != NULL && syncUtilUserPreCommit(pAppendEntry->originalRpcType)) { - SFsmCbMeta cbMeta = {0}; - cbMeta.index = pAppendEntry->index; - cbMeta.lastConfigIndex = syncNodeGetSnapshotConfigIndex(ths, cbMeta.index); - cbMeta.isWeak = pAppendEntry->isWeak; - cbMeta.code = 3; - cbMeta.state = ths->state; - cbMeta.seqNum = pAppendEntry->seqNum; - ths->pFsm->FpPreCommitCb(ths->pFsm, &rpcMsg, cbMeta); - } - } - rpcFreeCont(rpcMsg.pCont); + syncNodePreCommit(ths, pAppendEntry, 0); // free memory syncEntryDestory(pAppendEntry); @@ -440,7 +410,7 @@ static int32_t syncNodeDoMakeLogSame(SSyncNode* ths, SyncIndex FromIndex) { return code; } -static int32_t syncNodePreCommit(SSyncNode* ths, SSyncRaftEntry* pEntry) { +int32_t syncNodePreCommit(SSyncNode* ths, SSyncRaftEntry* pEntry, int32_t code) { SRpcMsg rpcMsg; syncEntry2OriginalRpc(pEntry, &rpcMsg); @@ -456,7 +426,7 @@ static int32_t syncNodePreCommit(SSyncNode* ths, SSyncRaftEntry* pEntry) { cbMeta.index = pEntry->index; cbMeta.lastConfigIndex = syncNodeGetSnapshotConfigIndex(ths, cbMeta.index); cbMeta.isWeak = pEntry->isWeak; - cbMeta.code = 2; + cbMeta.code = code; cbMeta.state = ths->state; cbMeta.seqNum = pEntry->seqNum; ths->pFsm->FpPreCommitCb(ths->pFsm, &rpcMsg, cbMeta); @@ -594,7 +564,7 @@ int32_t syncNodeOnAppendEntriesSnapshot2Cb(SSyncNode* ths, SyncAppendEntriesBatc return -1; } - code = syncNodePreCommit(ths, pAppendEntry); + code = syncNodePreCommit(ths, pAppendEntry, 0); ASSERT(code == 0); // syncEntryDestory(pAppendEntry); @@ -715,7 +685,7 @@ int32_t syncNodeOnAppendEntriesSnapshot2Cb(SSyncNode* ths, SyncAppendEntriesBatc return -1; } - code = syncNodePreCommit(ths, pAppendEntry); + code = syncNodePreCommit(ths, pAppendEntry, 0); ASSERT(code == 0); // syncEntryDestory(pAppendEntry); @@ -919,7 +889,7 @@ int32_t syncNodeOnAppendEntriesSnapshotCb(SSyncNode* ths, SyncAppendEntries* pMs } // pre commit - code = syncNodePreCommit(ths, pAppendEntry); + code = syncNodePreCommit(ths, pAppendEntry, 0); ASSERT(code == 0); // update match index @@ -1032,7 +1002,7 @@ int32_t syncNodeOnAppendEntriesSnapshotCb(SSyncNode* ths, SyncAppendEntries* pMs } // pre commit - code = syncNodePreCommit(ths, pAppendEntry); + code = syncNodePreCommit(ths, pAppendEntry, 0); ASSERT(code == 0); syncEntryDestory(pAppendEntry); diff --git a/source/libs/sync/src/syncCommit.c b/source/libs/sync/src/syncCommit.c index b3cdd079a49c6a13bc2b74c9d00f92e235d39ce8..fd6577477f171402f1769a02e18b60b765339e04 100644 --- a/source/libs/sync/src/syncCommit.c +++ b/source/libs/sync/src/syncCommit.c @@ -67,11 +67,6 @@ void syncMaybeAdvanceCommitIndex(SSyncNode* pSyncNode) { for (SyncIndex index = syncNodeGetLastIndex(pSyncNode); index > pSyncNode->commitIndex; --index) { bool agree = syncAgree(pSyncNode, index); - if (gRaftDetailLog) { - sTrace("syncMaybeAdvanceCommitIndex syncAgree:%d, index:%" PRId64 ", pSyncNode->commitIndex:%" PRId64, agree, - index, pSyncNode->commitIndex); - } - if (agree) { // term SSyncRaftEntry* pEntry = pSyncNode->pLogStore->getEntry(pSyncNode->pLogStore, index); @@ -82,20 +77,15 @@ void syncMaybeAdvanceCommitIndex(SSyncNode* pSyncNode) { // update commit index newCommitIndex = index; - if (gRaftDetailLog) { - sTrace("syncMaybeAdvanceCommitIndex maybe to update, newCommitIndex:%" PRId64 - " commit, pSyncNode->commitIndex:%" PRId64, - newCommitIndex, pSyncNode->commitIndex); - } - syncEntryDestory(pEntry); break; } else { - if (gRaftDetailLog) { - sTrace("syncMaybeAdvanceCommitIndex can not commit due to term not equal, pEntry->term:%" PRIu64 - ", pSyncNode->pRaftStore->currentTerm:%" PRIu64, - pEntry->term, pSyncNode->pRaftStore->currentTerm); - } + do { + char logBuf[128]; + snprintf(logBuf, sizeof(logBuf), "can not commit due to term not equal, index:%ld, term:%lu", pEntry->index, + pEntry->term); + syncNodeEventLog(pSyncNode, logBuf); + } while (0); } syncEntryDestory(pEntry); @@ -107,10 +97,6 @@ void syncMaybeAdvanceCommitIndex(SSyncNode* pSyncNode) { SyncIndex beginIndex = pSyncNode->commitIndex + 1; SyncIndex endIndex = newCommitIndex; - if (gRaftDetailLog) { - sTrace("syncMaybeAdvanceCommitIndex sync commit %" PRId64, newCommitIndex); - } - // update commit index pSyncNode->commitIndex = newCommitIndex; diff --git a/source/libs/sync/src/syncMain.c b/source/libs/sync/src/syncMain.c index a453b2572ce0a2b70ecf281d37a5537133f70343..935d89b99b336b5438892be7d5726c9c04bb4ee8 100644 --- a/source/libs/sync/src/syncMain.c +++ b/source/libs/sync/src/syncMain.c @@ -2504,22 +2504,7 @@ int32_t syncNodeOnClientRequestCb(SSyncNode* ths, SyncClientRequest* pMsg, SyncI } // pre commit - SRpcMsg rpcMsg; - syncEntry2OriginalRpc(pEntry, &rpcMsg); - - if (ths->pFsm != NULL) { - if (ths->pFsm->FpPreCommitCb != NULL && syncUtilUserPreCommit(pEntry->originalRpcType)) { - SFsmCbMeta cbMeta = {0}; - cbMeta.index = pEntry->index; - cbMeta.lastConfigIndex = syncNodeGetSnapshotConfigIndex(ths, cbMeta.index); - cbMeta.isWeak = pEntry->isWeak; - cbMeta.code = 0; - cbMeta.state = ths->state; - cbMeta.seqNum = pEntry->seqNum; - ths->pFsm->FpPreCommitCb(ths->pFsm, &rpcMsg, cbMeta); - } - } - rpcFreeCont(rpcMsg.pCont); + syncNodePreCommit(ths, pEntry, 0); // if only myself, maybe commit right now if (ths->replicaNum == 1) { @@ -2528,22 +2513,7 @@ int32_t syncNodeOnClientRequestCb(SSyncNode* ths, SyncClientRequest* pMsg, SyncI } else { // pre commit - SRpcMsg rpcMsg; - syncEntry2OriginalRpc(pEntry, &rpcMsg); - - if (ths->pFsm != NULL) { - if (ths->pFsm->FpPreCommitCb != NULL && syncUtilUserPreCommit(pEntry->originalRpcType)) { - SFsmCbMeta cbMeta = {0}; - cbMeta.index = pEntry->index; - cbMeta.lastConfigIndex = syncNodeGetSnapshotConfigIndex(ths, cbMeta.index); - cbMeta.isWeak = pEntry->isWeak; - cbMeta.code = 1; - cbMeta.state = ths->state; - cbMeta.seqNum = pEntry->seqNum; - ths->pFsm->FpPreCommitCb(ths->pFsm, &rpcMsg, cbMeta); - } - } - rpcFreeCont(rpcMsg.pCont); + syncNodePreCommit(ths, pEntry, 0); } if (pRetIndex != NULL) { diff --git a/source/libs/wal/src/walMgmt.c b/source/libs/wal/src/walMgmt.c index 85238e87b9f5c5549f2a1cf82435dca8fd85b5a4..047354c4aa44c5faa4bf50ec087ab939b8df0855 100644 --- a/source/libs/wal/src/walMgmt.c +++ b/source/libs/wal/src/walMgmt.c @@ -101,8 +101,8 @@ SWal *walOpen(const char *path, SWalCfg *pCfg) { // open meta walResetVer(&pWal->vers); - pWal->pWriteLogTFile = NULL; - pWal->pWriteIdxTFile = NULL; + pWal->pLogFile = NULL; + pWal->pIdxFile = NULL; pWal->writeCur = -1; pWal->fileInfoSet = taosArrayInit(8, sizeof(SWalFileInfo)); if (pWal->fileInfoSet == NULL) { @@ -179,10 +179,10 @@ int32_t walAlter(SWal *pWal, SWalCfg *pCfg) { void walClose(SWal *pWal) { taosThreadMutexLock(&pWal->mutex); - taosCloseFile(&pWal->pWriteLogTFile); - pWal->pWriteLogTFile = NULL; - taosCloseFile(&pWal->pWriteIdxTFile); - pWal->pWriteIdxTFile = NULL; + taosCloseFile(&pWal->pLogFile); + pWal->pLogFile = NULL; + taosCloseFile(&pWal->pIdxFile); + pWal->pIdxFile = NULL; walSaveMeta(pWal); taosArrayDestroy(pWal->fileInfoSet); pWal->fileInfoSet = NULL; @@ -223,7 +223,7 @@ static void walFsyncAll() { if (walNeedFsync(pWal)) { wTrace("vgId:%d, do fsync, level:%d seq:%d rseq:%d", pWal->cfg.vgId, pWal->cfg.level, pWal->fsyncSeq, atomic_load_32(&tsWal.seq)); - int32_t code = taosFsyncFile(pWal->pWriteLogTFile); + int32_t code = taosFsyncFile(pWal->pLogFile); if (code != 0) { wError("vgId:%d, file:%" PRId64 ".log, failed to fsync since %s", pWal->cfg.vgId, walGetLastFileFirstVer(pWal), strerror(code)); diff --git a/source/libs/wal/src/walSeek.c b/source/libs/wal/src/walSeek.c index 78d45c84e2699770bb3453521c72c0d9dfbc6ac9..87ab155065021cb902739ef18c8925afc04c2f07 100644 --- a/source/libs/wal/src/walSeek.c +++ b/source/libs/wal/src/walSeek.c @@ -22,8 +22,8 @@ static int64_t walSeekWritePos(SWal* pWal, int64_t ver) { int64_t code = 0; - TdFilePtr pIdxTFile = pWal->pWriteIdxTFile; - TdFilePtr pLogTFile = pWal->pWriteLogTFile; + TdFilePtr pIdxTFile = pWal->pIdxFile; + TdFilePtr pLogTFile = pWal->pLogFile; // seek position int64_t idxOff = walGetVerIdxOffset(pWal, ver); @@ -68,8 +68,8 @@ int walInitWriteFile(SWal* pWal) { return -1; } // switch file - pWal->pWriteIdxTFile = pIdxTFile; - pWal->pWriteLogTFile = pLogTFile; + pWal->pIdxFile = pIdxTFile; + pWal->pLogFile = pLogTFile; pWal->writeCur = taosArrayGetSize(pWal->fileInfoSet) - 1; return 0; } @@ -78,15 +78,15 @@ int walChangeWrite(SWal* pWal, int64_t ver) { int code; TdFilePtr pIdxTFile, pLogTFile; char fnameStr[WAL_FILE_LEN]; - if (pWal->pWriteLogTFile != NULL) { - code = taosCloseFile(&pWal->pWriteLogTFile); + if (pWal->pLogFile != NULL) { + code = taosCloseFile(&pWal->pLogFile); if (code != 0) { terrno = TAOS_SYSTEM_ERROR(errno); return -1; } } - if (pWal->pWriteIdxTFile != NULL) { - code = taosCloseFile(&pWal->pWriteIdxTFile); + if (pWal->pIdxFile != NULL) { + code = taosCloseFile(&pWal->pIdxFile); if (code != 0) { terrno = TAOS_SYSTEM_ERROR(errno); return -1; @@ -106,7 +106,7 @@ int walChangeWrite(SWal* pWal, int64_t ver) { pIdxTFile = taosOpenFile(fnameStr, TD_FILE_CREATE | TD_FILE_WRITE | TD_FILE_APPEND); if (pIdxTFile == NULL) { terrno = TAOS_SYSTEM_ERROR(errno); - pWal->pWriteIdxTFile = NULL; + pWal->pIdxFile = NULL; return -1; } walBuildLogName(pWal, fileFirstVer, fnameStr); @@ -114,12 +114,12 @@ int walChangeWrite(SWal* pWal, int64_t ver) { if (pLogTFile == NULL) { taosCloseFile(&pIdxTFile); terrno = TAOS_SYSTEM_ERROR(errno); - pWal->pWriteLogTFile = NULL; + pWal->pLogFile = NULL; return -1; } - pWal->pWriteLogTFile = pLogTFile; - pWal->pWriteIdxTFile = pIdxTFile; + pWal->pLogFile = pLogTFile; + pWal->pIdxFile = pIdxTFile; pWal->writeCur = idx; return fileFirstVer; } diff --git a/source/libs/wal/src/walWrite.c b/source/libs/wal/src/walWrite.c index 81500d80882f930e253771b835ac0f9e63f0f57b..d869e6e2ceee749657a53970c1981d97ca07db1a 100644 --- a/source/libs/wal/src/walWrite.c +++ b/source/libs/wal/src/walWrite.c @@ -32,8 +32,8 @@ int32_t walRestoreFromSnapshot(SWal *pWal, int64_t ver) { } } - taosCloseFile(&pWal->pWriteLogTFile); - taosCloseFile(&pWal->pWriteIdxTFile); + taosCloseFile(&pWal->pLogFile); + taosCloseFile(&pWal->pIdxFile); if (pWal->vers.firstVer != -1) { int32_t fileSetSize = taosArrayGetSize(pWal->fileInfoSet); @@ -261,14 +261,13 @@ int32_t walEndSnapshot(SWal *pWal) { pWal->vers.snapshotVer = ver; int ts = taosGetTimestampSec(); - int64_t minVerToDelete = ver; - void *pIter = NULL; + void *pIter = NULL; while (1) { pIter = taosHashIterate(pWal->pRefHash, pIter); if (pIter == NULL) break; SWalRef *pRef = *(SWalRef **)pIter; if (pRef->refVer == -1) continue; - minVerToDelete = TMIN(minVerToDelete, pRef->refVer); + ver = TMIN(ver, pRef->refVer); } int deleteCnt = 0; @@ -277,35 +276,37 @@ int32_t walEndSnapshot(SWal *pWal) { tmp.firstVer = ver; // find files safe to delete SWalFileInfo *pInfo = taosArraySearch(pWal->fileInfoSet, &tmp, compareWalFileInfo, TD_LE); - if (ver >= pInfo->lastVer) { - pInfo++; - } - // iterate files, until the searched result - for (SWalFileInfo *iter = pWal->fileInfoSet->pData; iter < pInfo; iter++) { - if ((pWal->cfg.retentionSize != -1 && newTotSize > pWal->cfg.retentionSize) || - (pWal->cfg.retentionPeriod != -1 && iter->closeTs + pWal->cfg.retentionPeriod > ts)) { - // delete according to file size or close time - deleteCnt++; - newTotSize -= iter->fileSize; + if (pInfo) { + if (ver >= pInfo->lastVer) { + pInfo++; + } + // iterate files, until the searched result + for (SWalFileInfo *iter = pWal->fileInfoSet->pData; iter < pInfo; iter++) { + if ((pWal->cfg.retentionSize != -1 && newTotSize > pWal->cfg.retentionSize) || + (pWal->cfg.retentionPeriod != -1 && iter->closeTs + pWal->cfg.retentionPeriod > ts)) { + // delete according to file size or close time + deleteCnt++; + newTotSize -= iter->fileSize; + } + } + char fnameStr[WAL_FILE_LEN]; + // remove file + for (int i = 0; i < deleteCnt; i++) { + pInfo = taosArrayGet(pWal->fileInfoSet, i); + walBuildLogName(pWal, pInfo->firstVer, fnameStr); + taosRemoveFile(fnameStr); + walBuildIdxName(pWal, pInfo->firstVer, fnameStr); + taosRemoveFile(fnameStr); } - } - char fnameStr[WAL_FILE_LEN]; - // remove file - for (int i = 0; i < deleteCnt; i++) { - pInfo = taosArrayGet(pWal->fileInfoSet, i); - walBuildLogName(pWal, pInfo->firstVer, fnameStr); - taosRemoveFile(fnameStr); - walBuildIdxName(pWal, pInfo->firstVer, fnameStr); - taosRemoveFile(fnameStr); - } - // make new array, remove files - taosArrayPopFrontBatch(pWal->fileInfoSet, deleteCnt); - if (taosArrayGetSize(pWal->fileInfoSet) == 0) { - pWal->writeCur = -1; - pWal->vers.firstVer = -1; - } else { - pWal->vers.firstVer = ((SWalFileInfo *)taosArrayGet(pWal->fileInfoSet, 0))->firstVer; + // make new array, remove files + taosArrayPopFrontBatch(pWal->fileInfoSet, deleteCnt); + if (taosArrayGetSize(pWal->fileInfoSet) == 0) { + pWal->writeCur = -1; + pWal->vers.firstVer = -1; + } else { + pWal->vers.firstVer = ((SWalFileInfo *)taosArrayGet(pWal->fileInfoSet, 0))->firstVer; + } } pWal->writeCur = taosArrayGetSize(pWal->fileInfoSet) - 1; pWal->totSize = newTotSize; @@ -324,34 +325,34 @@ END: int32_t walRollImpl(SWal *pWal) { int32_t code = 0; - if (pWal->pWriteIdxTFile != NULL) { - code = taosCloseFile(&pWal->pWriteIdxTFile); + if (pWal->pIdxFile != NULL) { + code = taosCloseFile(&pWal->pIdxFile); if (code != 0) { terrno = TAOS_SYSTEM_ERROR(errno); goto END; } } - if (pWal->pWriteLogTFile != NULL) { - code = taosCloseFile(&pWal->pWriteLogTFile); + if (pWal->pLogFile != NULL) { + code = taosCloseFile(&pWal->pLogFile); if (code != 0) { terrno = TAOS_SYSTEM_ERROR(errno); goto END; } } - TdFilePtr pIdxTFile, pLogTFile; + TdFilePtr pIdxFile, pLogFile; // create new file - int64_t newFileFirstVersion = pWal->vers.lastVer + 1; + int64_t newFileFirstVer = pWal->vers.lastVer + 1; char fnameStr[WAL_FILE_LEN]; - walBuildIdxName(pWal, newFileFirstVersion, fnameStr); - pIdxTFile = taosOpenFile(fnameStr, TD_FILE_CREATE | TD_FILE_WRITE | TD_FILE_APPEND); - if (pIdxTFile == NULL) { + walBuildIdxName(pWal, newFileFirstVer, fnameStr); + pIdxFile = taosOpenFile(fnameStr, TD_FILE_CREATE | TD_FILE_WRITE | TD_FILE_APPEND); + if (pIdxFile == NULL) { terrno = TAOS_SYSTEM_ERROR(errno); code = -1; goto END; } - walBuildLogName(pWal, newFileFirstVersion, fnameStr); - pLogTFile = taosOpenFile(fnameStr, TD_FILE_CREATE | TD_FILE_WRITE | TD_FILE_APPEND); - if (pLogTFile == NULL) { + walBuildLogName(pWal, newFileFirstVer, fnameStr); + pLogFile = taosOpenFile(fnameStr, TD_FILE_CREATE | TD_FILE_WRITE | TD_FILE_APPEND); + if (pLogFile == NULL) { terrno = TAOS_SYSTEM_ERROR(errno); code = -1; goto END; @@ -363,8 +364,8 @@ int32_t walRollImpl(SWal *pWal) { } // switch file - pWal->pWriteIdxTFile = pIdxTFile; - pWal->pWriteLogTFile = pLogTFile; + pWal->pIdxFile = pIdxFile; + pWal->pLogFile = pLogFile; pWal->writeCur = taosArrayGetSize(pWal->fileInfoSet) - 1; ASSERT(pWal->writeCur >= 0); @@ -378,10 +379,10 @@ END: static int32_t walWriteIndex(SWal *pWal, int64_t ver, int64_t offset) { SWalIdxEntry entry = {.ver = ver, .offset = offset}; - int64_t idxOffset = taosLSeekFile(pWal->pWriteIdxTFile, 0, SEEK_END); + int64_t idxOffset = taosLSeekFile(pWal->pIdxFile, 0, SEEK_END); wDebug("vgId:%d, write index, index:%" PRId64 ", offset:%" PRId64 ", at %" PRId64, pWal->cfg.vgId, ver, offset, idxOffset); - int64_t size = taosWriteFile(pWal->pWriteIdxTFile, &entry, sizeof(SWalIdxEntry)); + int64_t size = taosWriteFile(pWal->pIdxFile, &entry, sizeof(SWalIdxEntry)); if (size != sizeof(SWalIdxEntry)) { terrno = TAOS_SYSTEM_ERROR(errno); // TODO truncate @@ -407,7 +408,7 @@ static FORCE_INLINE int32_t walWriteImpl(SWal *pWal, int64_t index, tmsg_t msgTy pWal->writeHead.cksumHead = walCalcHeadCksum(&pWal->writeHead); pWal->writeHead.cksumBody = walCalcBodyCksum(body, bodyLen); - if (taosWriteFile(pWal->pWriteLogTFile, &pWal->writeHead, sizeof(SWalCkHead)) != sizeof(SWalCkHead)) { + if (taosWriteFile(pWal->pLogFile, &pWal->writeHead, sizeof(SWalCkHead)) != sizeof(SWalCkHead)) { // TODO ftruncate terrno = TAOS_SYSTEM_ERROR(errno); wError("vgId:%d, file:%" PRId64 ".log, failed to write since %s", pWal->cfg.vgId, walGetLastFileFirstVer(pWal), @@ -416,7 +417,7 @@ static FORCE_INLINE int32_t walWriteImpl(SWal *pWal, int64_t index, tmsg_t msgTy goto END; } - if (taosWriteFile(pWal->pWriteLogTFile, (char *)body, bodyLen) != bodyLen) { + if (taosWriteFile(pWal->pLogFile, (char *)body, bodyLen) != bodyLen) { // TODO ftruncate terrno = TAOS_SYSTEM_ERROR(errno); wError("vgId:%d, file:%" PRId64 ".log, failed to write since %s", pWal->cfg.vgId, walGetLastFileFirstVer(pWal), @@ -456,14 +457,14 @@ int64_t walAppendLog(SWal *pWal, tmsg_t msgType, SWalSyncInfo syncMeta, const vo return -1; } - if (pWal->pWriteIdxTFile == NULL || pWal->pWriteIdxTFile == NULL || pWal->writeCur < 0) { + if (pWal->pIdxFile == NULL || pWal->pIdxFile == NULL || pWal->writeCur < 0) { if (walInitWriteFile(pWal) < 0) { taosThreadMutexUnlock(&pWal->mutex); return -1; } } - ASSERT(pWal->pWriteIdxTFile != NULL && pWal->pWriteLogTFile != NULL && pWal->writeCur >= 0); + ASSERT(pWal->pIdxFile != NULL && pWal->pLogFile != NULL && pWal->writeCur >= 0); if (walWriteImpl(pWal, index, msgType, syncMeta, body, bodyLen) < 0) { taosThreadMutexUnlock(&pWal->mutex); @@ -494,14 +495,14 @@ int32_t walWriteWithSyncInfo(SWal *pWal, int64_t index, tmsg_t msgType, SWalSync return -1; } - if (pWal->pWriteIdxTFile == NULL || pWal->pWriteIdxTFile == NULL || pWal->writeCur < 0) { + if (pWal->pIdxFile == NULL || pWal->pIdxFile == NULL || pWal->writeCur < 0) { if (walInitWriteFile(pWal) < 0) { taosThreadMutexUnlock(&pWal->mutex); return -1; } } - ASSERT(pWal->pWriteIdxTFile != NULL && pWal->pWriteLogTFile != NULL && pWal->writeCur >= 0); + ASSERT(pWal->pIdxFile != NULL && pWal->pLogFile != NULL && pWal->writeCur >= 0); if (walWriteImpl(pWal, index, msgType, syncMeta, body, bodyLen) < 0) { taosThreadMutexUnlock(&pWal->mutex); @@ -524,7 +525,7 @@ int32_t walWrite(SWal *pWal, int64_t index, tmsg_t msgType, const void *body, in void walFsync(SWal *pWal, bool forceFsync) { if (forceFsync || (pWal->cfg.level == TAOS_WAL_FSYNC && pWal->cfg.fsyncPeriod == 0)) { wTrace("vgId:%d, fileId:%" PRId64 ".log, do fsync", pWal->cfg.vgId, walGetCurFileFirstVer(pWal)); - if (taosFsyncFile(pWal->pWriteLogTFile) < 0) { + if (taosFsyncFile(pWal->pLogFile) < 0) { wError("vgId:%d, file:%" PRId64 ".log, fsync failed since %s", pWal->cfg.vgId, walGetCurFileFirstVer(pWal), strerror(errno)); } diff --git a/tests/pytest/crash_gen/crash_gen_main.py b/tests/pytest/crash_gen/crash_gen_main.py index d5ffc1b7c1db1b86d238dffb1eb3b78724892ffe..7ab09383bfd0791aad30b39fa7e65ed66df9d936 100755 --- a/tests/pytest/crash_gen/crash_gen_main.py +++ b/tests/pytest/crash_gen/crash_gen_main.py @@ -371,7 +371,9 @@ class ThreadCoordinator: if isinstance(err, CrashGenError): # our own transition failure Logging.info("State transition error") # TODO: saw an error here once, let's print out stack info for err? - traceback.print_stack() + traceback.print_stack() # Stack frame to here. + Logging.info("Caused by:") + traceback.print_exception(*sys.exc_info()) # Ref: https://www.geeksforgeeks.org/how-to-print-exception-stack-trace-in-python/ transitionFailed = True self._te = None # Not running any more self._execStats.registerFailure("State transition error: {}".format(err)) @@ -741,7 +743,8 @@ class AnyState: sCnt += 1 if (sCnt >= 2): raise CrashGenError( - "Unexpected more than 1 success with task: {}, in task set: {}".format( + "Unexpected more than 1 success at state: {}, with task: {}, in task set: {}".format( + self.__class__.__name__, cls.__name__, # verified just now that isinstance(task, cls) [c.__class__.__name__ for c in tasks] )) @@ -756,8 +759,11 @@ class AnyState: if task.isSuccess(): sCnt += 1 if (exists and sCnt <= 0): - raise CrashGenError("Unexpected zero success for task type: {}, from tasks: {}" - .format(cls, tasks)) + raise CrashGenError("Unexpected zero success at state: {}, with task: {}, in task set: {}".format( + self.__class__.__name__, + cls.__name__, # verified just now that isinstance(task, cls) + [c.__class__.__name__ for c in tasks] + )) def assertNoTask(self, tasks, cls): for task in tasks: @@ -809,8 +815,6 @@ class StateEmpty(AnyState): ] def verifyTasksToState(self, tasks, newState): - if Config.getConfig().ignore_errors: # if we are asked to ignore certain errors, let's not verify CreateDB success. - return if (self.hasSuccess(tasks, TaskCreateDb) ): # at EMPTY, if there's succes in creating DB if (not self.hasTask(tasks, TaskDropDb)): # and no drop_db tasks @@ -995,16 +999,17 @@ class StateMechine: dbc.execute("show dnodes") # Generic Checks, first based on the start state - if self._curState.canCreateDb(): - self._curState.assertIfExistThenSuccess(tasks, TaskCreateDb) - # self.assertAtMostOneSuccess(tasks, CreateDbTask) # not really, in - # case of multiple creation and drops - - if self._curState.canDropDb(): - if gSvcMgr == None: # only if we are running as client-only - self._curState.assertIfExistThenSuccess(tasks, TaskDropDb) - # self.assertAtMostOneSuccess(tasks, DropDbTask) # not really in - # case of drop-create-drop + if not Config.getConfig().ignore_errors: # verify state, only if we are asked not to ignore certain errors. + if self._curState.canCreateDb(): + self._curState.assertIfExistThenSuccess(tasks, TaskCreateDb) + # self.assertAtMostOneSuccess(tasks, CreateDbTask) # not really, in + # case of multiple creation and drops + + if self._curState.canDropDb(): + if gSvcMgr == None: # only if we are running as client-only + self._curState.assertIfExistThenSuccess(tasks, TaskDropDb) + # self.assertAtMostOneSuccess(tasks, DropDbTask) # not really in + # case of drop-create-drop # if self._state.canCreateFixedTable(): # self.assertIfExistThenSuccess(tasks, CreateFixedTableTask) # Not true, DB may be dropped @@ -1026,7 +1031,8 @@ class StateMechine: newState = self._findCurrentState(dbc) Logging.debug("[STT] New DB state determined: {}".format(newState)) # can old state move to new state through the tasks? - self._curState.verifyTasksToState(tasks, newState) + if not Config.getConfig().ignore_errors: # verify state, only if we are asked not to ignore certain errors. + self._curState.verifyTasksToState(tasks, newState) self._curState = newState def pickTaskType(self): @@ -2231,16 +2237,14 @@ class TaskAddData(StateTransitionTask): class ThreadStacks: # stack info for all threads def __init__(self): self._allStacks = {} - allFrames = sys._current_frames() # All current stack frames + allFrames = sys._current_frames() # All current stack frames, keyed with "ident" for th in threading.enumerate(): # For each thread - if th.ident is None: - continue - stack = traceback.extract_stack(allFrames[th.ident]) # Get stack for a thread - shortTid = th.ident % 10000 + stack = traceback.extract_stack(allFrames[th.ident]) #type: ignore # Get stack for a thread + shortTid = th.native_id % 10000 #type: ignore self._allStacks[shortTid] = stack # Was using th.native_id def print(self, filteredEndName = None, filterInternal = False): - for tIdent, stack in self._allStacks.items(): # for each thread, stack frames top to bottom + for shortTid, stack in self._allStacks.items(): # for each thread, stack frames top to bottom lastFrame = stack[-1] if filteredEndName: # we need to filter out stacks that match this name if lastFrame.name == filteredEndName : # end did not match @@ -2252,7 +2256,9 @@ class ThreadStacks: # stack info for all threads '__init__']: # the thread that extracted the stack continue # ignore # Now print - print("\n<----- Thread Info for LWP/ID: {} (most recent call last) <-----".format(tIdent)) + print("\n<----- Thread Info for LWP/ID: {} (most recent call last) <-----".format(shortTid)) + lastSqlForThread = DbConn.fetchSqlForThread(shortTid) + print("Last SQL statement attempted from thread {} is: {}".format(shortTid, lastSqlForThread)) stackFrame = 0 for frame in stack: # was using: reversed(stack) # print(frame) diff --git a/tests/pytest/crash_gen/shared/db.py b/tests/pytest/crash_gen/shared/db.py index 75931ace48ed65708c7dfa97d01a426a0baa8203..6da0216d95cefe17512be8f119888bab1d14b7e8 100644 --- a/tests/pytest/crash_gen/shared/db.py +++ b/tests/pytest/crash_gen/shared/db.py @@ -27,6 +27,26 @@ class DbConn: TYPE_REST = "rest-api" TYPE_INVALID = "invalid" + # class variables + lastSqlFromThreads : dict[int, str] = {} # stored by thread id, obtained from threading.current_thread().ident%10000 + + @classmethod + def saveSqlForCurrentThread(cls, sql: str): + ''' + Let us save the last SQL statement on a per-thread basis, so that when later we + run into a dead-lock situation, we can pick out the deadlocked thread, and use + that information to find what what SQL statement is stuck. + ''' + th = threading.current_thread() + shortTid = th.native_id % 10000 #type: ignore + cls.lastSqlFromThreads[shortTid] = sql # Save this for later + + @classmethod + def fetchSqlForThread(cls, shortTid : int) -> str : + if shortTid not in cls.lastSqlFromThreads: + raise CrashGenError("No last-attempted-SQL found for thread id: {}".format(shortTid)) + return cls.lastSqlFromThreads[shortTid] + @classmethod def create(cls, connType, dbTarget): if connType == cls.TYPE_NATIVE: @@ -163,6 +183,7 @@ class DbConnRest(DbConn): def _doSql(self, sql): self._lastSql = sql # remember this, last SQL attempted + self.saveSqlForCurrentThread(sql) # Save in global structure too. #TODO: combine with above try: r = requests.post(self._url, data = sql, @@ -392,6 +413,7 @@ class DbConnNative(DbConn): "Cannot exec SQL unless db connection is open", CrashGenError.DB_CONNECTION_NOT_OPEN) Logging.debug("[SQL] Executing SQL: {}".format(sql)) self._lastSql = sql + self.saveSqlForCurrentThread(sql) # Save in global structure too. #TODO: combine with above nRows = self._tdSql.execute(sql) cls = self.__class__ cls.totalRequests += 1 @@ -407,6 +429,7 @@ class DbConnNative(DbConn): "Cannot query database until connection is open, restarting?", CrashGenError.DB_CONNECTION_NOT_OPEN) Logging.debug("[SQL] Executing SQL: {}".format(sql)) self._lastSql = sql + self.saveSqlForCurrentThread(sql) # Save in global structure too. #TODO: combine with above nRows = self._tdSql.query(sql) cls = self.__class__ cls.totalRequests += 1 diff --git a/tests/script/jenkins/basic.txt b/tests/script/jenkins/basic.txt index a606311f3c3676141a401b891eb142335244a6ee..0c19e4a2fe06f477f3e52eb4fcecf0541ba181d5 100644 --- a/tests/script/jenkins/basic.txt +++ b/tests/script/jenkins/basic.txt @@ -224,7 +224,7 @@ # ---- stream ./test.sh -f tsim/stream/basic0.sim -#./test.sh -f tsim/stream/basic1.sim +./test.sh -f tsim/stream/basic1.sim ./test.sh -f tsim/stream/basic2.sim ./test.sh -f tsim/stream/drop_stream.sim ./test.sh -f tsim/stream/distributeInterval0.sim diff --git a/tests/script/tmp/data.sim b/tests/script/tmp/data.sim index dcfa02e0a73dd62469b6c88f2d49865a1eba0439..f43987ffcb5f5be36774a75da6dd1dea975dcbf3 100644 --- a/tests/script/tmp/data.sim +++ b/tests/script/tmp/data.sim @@ -3,6 +3,7 @@ system sh/deploy.sh -n dnode1 -i 1 system sh/deploy.sh -n dnode2 -i 2 system sh/deploy.sh -n dnode3 -i 3 system sh/deploy.sh -n dnode4 -i 4 +system sh/cfg.sh -n dnode1 -c supportVnodes -v 0 system sh/exec.sh -n dnode1 -s start system sh/exec.sh -n dnode2 -s start system sh/exec.sh -n dnode3 -s start @@ -44,6 +45,8 @@ if $data(4)[4] != ready then goto step1 endi +return + print =============== step2: create database sql create database db vgroups 1 replica 3 sql show databases diff --git a/tests/script/tsim/parser/select_with_tags.sim b/tests/script/tsim/parser/select_with_tags.sim index 5130b39f487fcedacdae699cc785d3fc1ef260b4..7a2c1217e97e4993a86a4ee19dc52911268c725c 100644 --- a/tests/script/tsim/parser/select_with_tags.sim +++ b/tests/script/tsim/parser/select_with_tags.sim @@ -183,7 +183,7 @@ if $rows != 12800 then return -1 endi -sql select _rowts, top(c1, 80), tbname, t1, t2 from select_tags_mt0; +sql select ts, top(c1, 80), tbname, t1, t2 from select_tags_mt0 order by ts; if $rows != 80 then return -1 endi @@ -212,7 +212,7 @@ if $data04 != @abc12@ then return -1 endi -sql select top(c1, 80), tbname, t1, t2 from select_tags_mt0; +sql select ts, top(c1, 80), tbname, t1, t2 from select_tags_mt0 order by ts; if $rows != 80 then return -1 endi @@ -241,7 +241,7 @@ if $data04 != @abc12@ then return -1 endi -sql select bottom(c1, 72), tbname, t1, t2 from select_tags_mt0; +sql select ts, bottom(c1, 72), tbname, t1, t2 from select_tags_mt0 order by ts; if $rows != 72 then return -1 endi @@ -293,7 +293,7 @@ if $data03 != 15 then endi print ====== selectivity+tags+group by tags======================= -sql select first(c1), tbname, t1, t2 from select_tags_mt0 group by tbname; +sql select first(c1), tbname, t1, t2, tbname from select_tags_mt0 group by tbname order by t1; if $rows != 16 then return -1 endi @@ -327,7 +327,7 @@ if $data04 != @select_tags_tb0@ then return -1 endi -sql select last_row(ts,c1), tbname, t1, t2 from select_tags_mt0 group by tbname; +sql select last_row(ts,c1), tbname, t1, t2, tbname from select_tags_mt0 group by tbname order by t1; if $rows != 16 then return -1 endi @@ -361,7 +361,7 @@ if $data04 != @abc0@ then return -1 endi -sql select tbname,t1,t2 from select_tags_mt0; +sql select distinct tbname,t1,t2 from select_tags_mt0; if $row != 16 then return -1 endi @@ -411,7 +411,7 @@ if $data11 != @70-01-01 08:01:40.001@ then return -1 endi -sql select top(c1, 100), tbname, t1, t2 from select_tags_mt0 where tbname in ('select_tags_tb0', 'select_tags_tb1') group by tbname; +sql select ts, top(c1, 100), tbname, t1, t2 from select_tags_mt0 where tbname in ('select_tags_tb0', 'select_tags_tb1') group by tbname order by ts; if $row != 200 then return -1 endi @@ -448,7 +448,7 @@ if $data04 != @abc0@ then return -1 endi -sql select top(c1, 2), t2 from select_tags_mt0 where tbname in ('select_tags_tb0', 'select_tags_tb1') group by tbname,t2; +sql select ts, top(c1, 2), t2, tbname, t2 from select_tags_mt0 where tbname in ('select_tags_tb0', 'select_tags_tb1') group by tbname,t2 order by ts; if $row != 4 then return -1 endi @@ -535,33 +535,13 @@ endi # slimit /limit -sql select top(c1, 2), t2 from select_tags_mt0 where tbname in ('select_tags_tb0', 'select_tags_tb1') group by tbname,t2 limit 2 offset 1; +sql select ts, top(c1, 2), t2 from select_tags_mt0 where tbname in ('select_tags_tb0', 'select_tags_tb1') group by tbname,t2 limit 2 offset 1; if $row != 2 then return -1 endi -if $data00 != @70-01-01 08:01:40.199@ then - return -1 -endi - -if $data01 != 99 then - return -1 -endi - -if $data02 != @abc0@ then - return -1 -endi - -if $data03 != @select_tags_tb0@ then - return -1 -endi - -if $data04 != @abc0@ then - return -1 -endi - print ======= selectivity + tags + group by + tags + filter =========================== -sql select first(c1), t1 from select_tags_mt0 where c1<=2 group by tbname; +sql select first(c1), t1, tbname from select_tags_mt0 where c1<=2 group by tbname order by t1; if $row != 3 then return -1 endi @@ -602,7 +582,7 @@ if $data22 != @select_tags_tb2@ then return -1 endi -sql select first(c1), tbname from select_tags_mt0 where c1<=2 interval(1s); +sql select _wstart, first(c1), tbname from select_tags_mt0 where c1<=2 interval(1s); if $row != 3 then return -1 endi @@ -671,7 +651,7 @@ if $data01 != @70-01-01 08:01:50.001@ then endi print ======= selectivity + tags + group by + tags + filter + interval ================ -sql select first(c1), t2, t1, tbname from select_tags_mt0 where c1<=2 interval(1d) group by tbname; +sql select _wstart,first(c1), t2, t1, tbname, tbname from select_tags_mt0 where c1<=2 partition by tbname interval(1d) order by t1; if $row != 3 then return -1 endi @@ -708,7 +688,7 @@ if $data25 != @select_tags_tb2@ then return -1 endi -sql select top(c1, 5), t2 from select_tags_mt0 where c1<=2 interval(1d) group by tbname; +sql select ts, top(c1, 5), t2, tbname from select_tags_mt0 where c1<=2 partition by tbname interval(1d) order by ts, t2; if $row != 15 then return -1 endi @@ -746,7 +726,7 @@ if $data93 != @select_tags_tb1@ then endi #if data -sql select top(c1, 50), t2, t1, tbname from select_tags_mt0 where c1<=2 interval(1d) group by tbname; +sql select ts, top(c1, 50), t2, t1, tbname, tbname from select_tags_mt0 where c1<=2 partition by tbname interval(1d) order by ts, t2; if $row != 48 then return -1 endi @@ -831,7 +811,7 @@ endi print TODO ======= selectivity + tags+ group by + tags + filter + interval + join=========== print ==========================mix tag columns and group by columns====================== -sql select top(c1, 100), tbname from select_tags_mt0 where tbname in ('select_tags_tb0', 'select_tags_tb1') group by t3 +sql select ts, top(c1, 100), tbname, t3 from select_tags_mt0 where tbname in ('select_tags_tb0', 'select_tags_tb1') group by t3 order by ts, tbname; if $rows != 100 then return -1 endi @@ -887,9 +867,9 @@ sql_error select twa(c2), tbname from select_tags_mt0; sql_error select interp(c2), tbname from select_tags_mt0 where ts=100001; sql_error select t1,t2,tbname from select_tags_mt0 group by tbname; -sql_error select count(tbname) from select_tags_mt0 interval(1d); -sql_error select count(tbname) from select_tags_mt0 group by t1; -sql_error select count(tbname),SUM(T1) from select_tags_mt0 interval(1d); +sql select count(tbname) from select_tags_mt0 interval(1d); +sql select count(tbname) from select_tags_mt0 group by t1; +sql select count(tbname),SUM(T1) from select_tags_mt0 interval(1d); sql_error select first(c1), count(*), t2, t1, tbname from select_tags_mt0 where c1<=2 interval(1d) group by tbname; sql_error select ts from select_tags_mt0 interval(1y); sql_error select count(*), tbname from select_tags_mt0 interval(1y); @@ -902,8 +882,8 @@ sql_error select tbname, t1 from select_tags_mt0 interval(1y); #valid sql: select first(c1), tbname, t1 from select_tags_mt0 group by t2; print ==================================>TD-4231 -sql_error select t1,tbname from select_tags_mt0 where c1<0 -sql_error select t1,tbname from select_tags_mt0 where c1<0 and tbname in ('select_tags_tb12') +sql select t1,tbname from select_tags_mt0 where c1<0 +sql select t1,tbname from select_tags_mt0 where c1<0 and tbname in ('select_tags_tb12') sql select tbname from select_tags_mt0 where tbname in ('select_tags_tb12'); diff --git a/tests/script/tsim/valgrind/basic1.sim b/tests/script/tsim/valgrind/basic1.sim index f3d418cfd16da10ed15dc39ccb99ea8ac430446d..49f0b20702ca8cb86b1c8948896da8530691b8d8 100644 --- a/tests/script/tsim/valgrind/basic1.sim +++ b/tests/script/tsim/valgrind/basic1.sim @@ -1,6 +1,5 @@ system sh/stop_dnodes.sh system sh/deploy.sh -n dnode1 -i 1 -system sh/cfg.sh -n dnode1 -c debugflag -v 131 system sh/exec.sh -n dnode1 -s start -v sql connect @@ -22,78 +21,31 @@ if $data(1)[4] != ready then goto step1 endi -print =============== step2: create db -sql create database db -sql use db -sql create table db.stb (ts timestamp, c1 int, c2 binary(4)) tags(t1 int, t2 float, t3 binary(16)) comment "abd" -sql create table db.c1 using db.stb tags(101, 102, "103") - -print =============== step3: alter stb -sql_error alter table db.stb add column ts int -sql alter table db.stb add column c3 int -sql alter table db.stb add column c4 bigint -sql alter table db.stb add column c5 binary(12) -sql alter table db.stb drop column c1 -sql alter table db.stb drop column c4 -sql alter table db.stb MODIFY column c2 binary(32) -sql alter table db.stb add tag t4 bigint -sql alter table db.stb add tag c1 int -sql alter table db.stb add tag t5 binary(12) -sql alter table db.stb drop tag c1 -sql alter table db.stb drop tag t5 -sql alter table db.stb MODIFY tag t3 binary(32) -sql alter table db.stb rename tag t1 tx -sql alter table db.stb comment 'abcde' ; -sql drop table db.stb - -print =============== step4: alter tb -sql create table tb (ts timestamp, a int) -sql insert into tb values(now-28d, -28) -sql select count(a) from tb -sql alter table tb add column b smallint -sql insert into tb values(now-25d, -25, 0) -sql select count(b) from tb -sql alter table tb add column c tinyint -sql insert into tb values(now-22d, -22, 3, 0) -sql select count(c) from tb -sql alter table tb add column d int -sql insert into tb values(now-19d, -19, 6, 0, 0) -sql select count(d) from tb -sql alter table tb add column e bigint -sql alter table tb add column f float -sql alter table tb add column g double -sql alter table tb add column h binary(10) -sql select count(a), count(b), count(c), count(d), count(e), count(f), count(g), count(h) from tb -sql select * from tb order by ts desc - -print =============== step5: alter stb and insert data -sql create table stb (ts timestamp, c1 int, c2 binary(4)) tags(t1 int, t2 float, t3 binary(16)) comment "abd" -sql show db.stables -sql describe stb -sql_error alter table stb add column ts int +$tbPrefix = tb +$tbNum = 5 +$rowNum = 10 -sql create table db.ctb using db.stb tags(101, 102, "103") -sql insert into db.ctb values(now, 1, "2") -sql show db.tables -sql select * from db.stb -sql select * from tb - -sql alter table stb add column c3 int -sql describe stb -sql select * from db.stb -sql select * from tb -sql insert into db.ctb values(now+1s, 1, 2, 3) -sql select * from db.stb - -sql alter table db.stb add column c4 bigint -sql select * from db.stb -sql insert into db.ctb values(now+2s, 1, 2, 3, 4) - -sql alter table db.stb drop column c1 -sql reset query cache -sql select * from tb -sql insert into db.ctb values(now+3s, 2, 3, 4) -sql select * from db.stb +print =============== step2: prepare data +sql create database db vgroups 2 +sql use db +sql create table if not exists stb (ts timestamp, tbcol int, tbcol2 float, tbcol3 double) tags (tgcol int unsigned) + +$i = 0 +while $i < $tbNum + $tb = $tbPrefix . $i + sql create table $tb using stb tags( $i ) + $x = 0 + while $x < $rowNum + $cc = $x * 60000 + $ms = 1601481600000 + $cc + sql insert into $tb values ($ms , $x , $x , $x ) + $x = $x + 1 + endw + $i = $i + 1 +endw + +print =============== step3: tb +sql select count(1) from tb1 _OVER: system sh/exec.sh -n dnode1 -s stop -x SIGINT diff --git a/tests/script/tsim/valgrind/basic2.sim b/tests/script/tsim/valgrind/basic2.sim index 45ac78daf002e079c0c995adf043d81e009dcea3..7c905209ee2967b458a7036c46ea7a1014b029ec 100644 --- a/tests/script/tsim/valgrind/basic2.sim +++ b/tests/script/tsim/valgrind/basic2.sim @@ -1,6 +1,7 @@ system sh/stop_dnodes.sh system sh/deploy.sh -n dnode1 -i 1 -system sh/exec.sh -n dnode1 -s start -v +system sh/cfg.sh -n dnode1 -c debugflag -v 131 +system sh/exec.sh -n dnode1 -s start sql connect print =============== step1: create drop show dnodes @@ -21,88 +22,73 @@ if $data(1)[4] != ready then goto step1 endi -print =============== step2: create db -sql create database db -sql use db -sql create table db.stb (ts timestamp, c1 int, c2 binary(4)) tags(t1 int, t2 float, t3 binary(16)) comment "abd" -sql create table db.c1 using db.stb tags(101, 102, "103") - -print =============== step3: alter stb -sql_error alter table db.stb add column ts int -sql alter table db.stb add column c3 int -sql alter table db.stb add column c4 bigint -sql alter table db.stb add column c5 binary(12) -sql alter table db.stb drop column c1 -sql alter table db.stb drop column c4 -sql alter table db.stb MODIFY column c2 binary(32) -sql alter table db.stb add tag t4 bigint -sql alter table db.stb add tag c1 int -sql alter table db.stb add tag t5 binary(12) -sql alter table db.stb drop tag c1 -sql alter table db.stb drop tag t5 -sql alter table db.stb MODIFY tag t3 binary(32) -sql alter table db.stb rename tag t1 tx -sql alter table db.stb comment 'abcde' ; -sql drop table db.stb - -print =============== step4: alter tb -sql create table tb (ts timestamp, a int) -sql insert into tb values(now-28d, -28) -sql select count(a) from tb -sql alter table tb add column b smallint -sql insert into tb values(now-25d, -25, 0) -sql select count(b) from tb -sql alter table tb add column c tinyint -sql insert into tb values(now-22d, -22, 3, 0) -sql select count(c) from tb -sql alter table tb add column d int -sql insert into tb values(now-19d, -19, 6, 0, 0) -sql select count(d) from tb -sql alter table tb add column e bigint -sql alter table tb add column f float -sql alter table tb add column g double -sql alter table tb add column h binary(10) -sql select count(a), count(b), count(c), count(d), count(e), count(f), count(g), count(h) from tb -sql select * from tb order by ts desc +$tbPrefix = tb +$tbNum = 5 +$rowNum = 10 -print =============== step5: alter stb and insert data -sql create table stb (ts timestamp, c1 int, c2 binary(4)) tags(t1 int, t2 float, t3 binary(16)) comment "abd" -sql show db.stables -sql describe stb -sql_error alter table stb add column ts int - -sql create table db.ctb using db.stb tags(101, 102, "103") -sql insert into db.ctb values(now, 1, "2") -sql show db.tables -sql select * from db.stb -sql select * from tb +print =============== step2: prepare data +sql create database db vgroups 2 +sql use db +sql create table if not exists stb (ts timestamp, tbcol int, tbcol2 float, tbcol3 double) tags (tgcol int unsigned) -sql alter table stb add column c3 int -sql describe stb -sql select * from db.stb -sql select * from tb -sql insert into db.ctb values(now+1s, 1, 2, 3) -sql select * from db.stb +$i = 0 +while $i < $tbNum + $tb = $tbPrefix . $i + sql create table $tb using stb tags( $i ) + $x = 0 + while $x < $rowNum + $cc = $x * 60000 + $ms = 1601481600000 + $cc + sql insert into $tb values ($ms , $x , $x , $x ) + $x = $x + 1 + endw -sql alter table db.stb add column c4 bigint -sql select * from db.stb -sql insert into db.ctb values(now+2s, 1, 2, 3, 4) + $cc = $x * 60000 + $ms = 1601481600000 + $cc + sql insert into $tb values ($ms , NULL , NULL , NULL ) + $i = $i + 1 +endw -sql alter table db.stb drop column c1 -sql reset query cache -sql select * from tb -sql insert into db.ctb values(now+3s, 2, 3, 4) -sql select * from db.stb +system sh/exec.sh -n dnode1 -s stop -x SIGINT +system sh/exec.sh -n dnode1 -s start -v -sql alter table db.stb add tag t4 bigint -sql select * from db.stb -sql select * from db.stb -sql_error create table db.ctb2 using db.stb tags(101, "102") -sql create table db.ctb2 using db.stb tags(101, 102, "103", 104) -sql insert into db.ctb2 values(now, 1, 2, 3) +print =============== step3: tb +sql select avg(tbcol) from tb1 +sql select avg(tbcol) from tb1 where ts <= 1601481840000 +sql select avg(tbcol) as b from tb1 +sql select avg(tbcol) as b from tb1 interval(1d) +sql select avg(tbcol) as b from tb1 where ts <= 1601481840000 interval(1m) +sql select bottom(tbcol, 2) from tb1 where ts <= 1601481840000 +sql select top(tbcol, 2) from tb1 where ts <= 1601481840000 +sql select percentile(tbcol, 2) from tb1 where ts <= 1601481840000 +sql select leastsquares(tbcol, 1, 1) as b from tb1 where ts <= 1601481840000 +sql show table distributed tb1 +sql select count(tbcol) as b from tb1 where ts <= 1601481840000 interval(1m) +sql select diff(tbcol) from tb1 where ts <= 1601481840000 +sql select diff(tbcol) from tb1 where tbcol > 5 and tbcol < 20 +sql select first(tbcol), last(tbcol) as b from tb1 where ts <= 1601481840000 interval(1m) +sql select count(tbcol), avg(tbcol), max(tbcol), min(tbcol), sum(tbcol), stddev(tbcol) from tb1 where ts <= 1601481840000 partition by tgcol interval(1m) +#sql select count(tbcol), avg(tbcol), max(tbcol), min(tbcol), count(tbcol) from tb1 where ts <= 1601481840000 and ts >= 1601481800000 partition by tgcol interval(1m) fill(value, 0) +sql select last_row(*) from tb1 where tbcol > 5 and tbcol < 20 -print =============== step6: query data -sql select * from db.stb where tbname = 'ctb2'; +print =============== step4: stb +sql select avg(tbcol) as c from stb +sql select avg(tbcol) as c from stb where ts <= 1601481840000 +sql select avg(tbcol) as c from stb where tgcol < 5 and ts <= 1601481840000 +sql select avg(tbcol) as c from stb interval(1m) +sql select avg(tbcol) as c from stb interval(1d) +sql select avg(tbcol) as b from stb where ts <= 1601481840000 interval(1m) +sql select avg(tbcol) as c from stb group by tgcol +sql select avg(tbcol) as b from stb where ts <= 1601481840000 partition by tgcol interval(1m) +sql show table distributed stb +sql select count(tbcol) as b from stb where ts <= 1601481840000 partition by tgcol interval(1m) +sql select diff(tbcol) from stb where ts <= 1601481840000 +sql select first(tbcol), last(tbcol) as c from stb group by tgcol +sql select first(tbcol), last(tbcol) as b from stb where ts <= 1601481840000 and tbcol2 is null partition by tgcol interval(1m) +sql select first(tbcol), last(tbcol) as b from stb where ts <= 1601481840000 partition by tgcol interval(1m) +sql select count(tbcol), avg(tbcol), max(tbcol), min(tbcol), sum(tbcol), stddev(tbcol) from stb where ts <= 1601481840000 partition by tgcol interval(1m) +#sql select count(tbcol), avg(tbcol), max(tbcol), min(tbcol), count(tbcol) from stb where ts <= 1601481840000 and ts >= 1601481800000 partition by tgcol interval(1m) fill(value, 0) +sql select last_row(tbcol), stddev(tbcol) from stb where tbcol > 5 and tbcol < 20 group by tgcol _OVER: system sh/exec.sh -n dnode1 -s stop -x SIGINT diff --git a/tests/script/tsim/valgrind/basic3.sim b/tests/script/tsim/valgrind/basic3.sim index d513eee3cf111072c0dc570d6afd4af74b28764f..b9ed1641c89eb855e231dc2f9f662613bcf742b9 100644 --- a/tests/script/tsim/valgrind/basic3.sim +++ b/tests/script/tsim/valgrind/basic3.sim @@ -1,5 +1,6 @@ system sh/stop_dnodes.sh system sh/deploy.sh -n dnode1 -i 1 +system sh/cfg.sh -n dnode1 -c debugflag -v 131 system sh/exec.sh -n dnode1 -s start -v sql connect @@ -44,20 +45,11 @@ while $i < $tbNum $i = $i + 1 endw -print =============== step3: avg -sql select avg(tbcol) from tb1 -sql select avg(tbcol) from tb1 where ts <= 1601481840000 -sql select avg(tbcol) as b from tb1 -sql select avg(tbcol) as b from tb1 interval(1d) -sql select avg(tbcol) as b from tb1 where ts <= 1601481840000s interval(1m) -sql select avg(tbcol) as c from stb -sql select avg(tbcol) as c from stb where ts <= 1601481840000 -sql select avg(tbcol) as c from stb where tgcol < 5 and ts <= 1601481840000 -sql select avg(tbcol) as c from stb interval(1m) -sql select avg(tbcol) as c from stb interval(1d) -sql select avg(tbcol) as b from stb where ts <= 1601481840000s interval(1m) -sql select avg(tbcol) as c from stb group by tgcol -sql select avg(tbcol) as b from stb where ts <= 1601481840000s partition by tgcol interval(1m) +print =============== step3: tb +sql select count(tbcol), avg(tbcol), max(tbcol), min(tbcol), count(tbcol) from tb1 where ts <= 1601481840000 and ts >= 1601481800000 partition by tgcol interval(1m) fill(value, 0) + +print =============== step4: stb +sql select count(tbcol), avg(tbcol), max(tbcol), min(tbcol), count(tbcol) from stb where ts <= 1601481840000 and ts >= 1601481800000 partition by tgcol interval(1m) fill(value, 0) _OVER: system sh/exec.sh -n dnode1 -s stop -x SIGINT diff --git a/tests/script/tsim/valgrind/basic4.sim b/tests/script/tsim/valgrind/basic4.sim new file mode 100644 index 0000000000000000000000000000000000000000..8be96f769b0cf265f646e88d37de312c2ccb20d0 --- /dev/null +++ b/tests/script/tsim/valgrind/basic4.sim @@ -0,0 +1,74 @@ +system sh/stop_dnodes.sh +system sh/deploy.sh -n dnode1 -i 1 +system sh/cfg.sh -n dnode1 -c debugflag -v 131 +system sh/exec.sh -n dnode1 -s start -v +sql connect + +print =============== step1: create drop show dnodes +$x = 0 +step1: + $x = $x + 1 + sleep 1000 + if $x == 10 then + print ---> dnode not ready! + return -1 + endi +sql show dnodes +print ---> $data00 $data01 $data02 $data03 $data04 $data05 +if $rows != 1 then + return -1 +endi +if $data(1)[4] != ready then + goto step1 +endi + +print =============== step2: create db +sql create database d1 vgroups 2 buffer 3 +sql show databases +sql use d1 +sql show vgroups + +print =============== step3: create show stable +sql create table if not exists stb (ts timestamp, c1 int, c2 float, c3 double) tags (t1 int unsigned) +sql show stables +if $rows != 1 then + return -1 +endi + +print =============== step4: create show table +sql create table ct1 using stb tags(1000) +sql create table ct2 using stb tags(2000) +sql create table ct3 using stb tags(3000) +sql show tables +if $rows != 3 then + return -1 +endi + +print =============== step5: insert data (null / update) +sql insert into ct1 values(now+0s, 10, 2.0, 3.0) +sql insert into ct1 values(now+1s, 11, 2.1, NULL)(now+2s, -12, -2.2, -3.2)(now+3s, -13, -2.3, -3.3) +sql insert into ct2 values(now+0s, 10, 2.0, 3.0) +sql insert into ct2 values(now+1s, 11, 2.1, 3.1)(now+2s, -12, -2.2, -3.2)(now+3s, -13, -2.3, -3.3) +sql insert into ct3 values('2021-01-01 00:00:00.000', NULL, NULL, 3.0) +sql insert into ct3 values('2022-03-02 16:59:00.010', 3 , 4, 5), ('2022-03-02 16:59:00.010', 33 , 4, 5), ('2022-04-01 16:59:00.011', 4, 4, 5), ('2022-04-01 16:59:00.011', 6, 4, 5), ('2022-03-06 16:59:00.013', 8, 4, 5); +sql insert into ct3 values('2022-03-02 16:59:00.010', 103, 1, 2), ('2022-03-02 16:59:00.010', 303, 3, 4), ('2022-04-01 16:59:00.011', 40, 5, 6), ('2022-04-01 16:59:00.011', 60, 4, 5), ('2022-03-06 16:59:00.013', 80, 4, 5); + +print =============== step6: query data= + +sql select * from stb where t1 between 1000 and 2500 + + +_OVER: +system sh/exec.sh -n dnode1 -s stop -x SIGINT +print =============== check +$null= + +system_content sh/checkValgrind.sh -n dnode1 +print cmd return result ----> [ $system_content ] +if $system_content > 0 then + return -1 +endi + +if $system_content == $null then + return -1 +endi diff --git a/tests/script/tsim/valgrind/checkError3.sim b/tests/script/tsim/valgrind/checkError3.sim index e8b25098d65f7252292a33be0accd69672fdc359..41623896b3b93dc84b8348f39feb2ff5ef20e964 100644 --- a/tests/script/tsim/valgrind/checkError3.sim +++ b/tests/script/tsim/valgrind/checkError3.sim @@ -37,6 +37,7 @@ sql show stables if $rows != 4 then return -1 endi +sql show stables like 'stb' print =============== step4: ccreate child table sql create table c1 using stb tags(true, -1, -2, -3, -4, -6.0, -7.0, 'child tbl 1', 'child tbl 1', '2022-02-25 18:00:00.000', 10, 20, 30, 40) diff --git a/tests/script/tsim/valgrind/checkError5.sim b/tests/script/tsim/valgrind/checkError5.sim index 6eef185fd3eec169f9556d196ef0f8d39094300f..f0786587d9597c14971f9d49ce4144014aef6d81 100644 --- a/tests/script/tsim/valgrind/checkError5.sim +++ b/tests/script/tsim/valgrind/checkError5.sim @@ -105,6 +105,39 @@ sql insert into db.ctb2 values(now, 1, 2, 3) print =============== step6: query data sql select * from db.stb where tbname = 'ctb2'; + +print =============== step7: normal table +sql create database d1 replica 1 duration 7 keep 50 +sql use d1 +sql create table tb (ts timestamp, a int) +sql insert into tb values(now-28d, -28) +sql alter table tb add column b smallint +sql insert into tb values(now-25d, -25, 0) +sql alter table tb add column c tinyint +sql insert into tb values(now-22d, -22, 3, 0) +sql alter table tb add column d int +sql insert into tb values(now-19d, -19, 6, 0, 0) +sql alter table tb add column e bigint +sql insert into tb values(now-16d, -16, 9, 0, 0, 0) +sql alter table tb add column f float +sql insert into tb values(now-13d, -13, 12, 0, 0, 0, 0) +sql alter table tb add column g double +sql insert into tb values(now-10d, -10, 15, 0, 0, 0, 0, 0) +sql alter table tb add column h binary(10) +sql insert into tb values(now-7d, -7, 18, 0, 0, 0, 0, 0, '0') +sql select count(a), count(b), count(c), count(d), count(e), count(f), count(g), count(h) from d1.tb; +sql alter table tb drop column a +sql insert into tb values(now-4d, 1, 1, 1, 1, 1, 1, '1') +sql alter table tb drop column b +sql insert into tb values(now-3d, 1, 1, 1, 1, 1, '1') +sql alter table tb drop column c +sql insert into tb values(now-2d, 1, 1, 1, 1, '1') +sql alter table tb drop column d +sql insert into tb values(now-1d, 1, 1, 1, '1') +sql alter table tb drop column e +sql insert into tb values(now, 1, 1, '1') +sql select count(h) from tb + _OVER: system sh/exec.sh -n dnode1 -s stop -x SIGINT print =============== check diff --git a/tests/script/tsim/valgrind/checkError6.sim b/tests/script/tsim/valgrind/checkError6.sim index 4681345839981624732f1af37864c40beb46218a..27fd291f6436593d21fa6a4c0ae085d392ba64d9 100644 --- a/tests/script/tsim/valgrind/checkError6.sim +++ b/tests/script/tsim/valgrind/checkError6.sim @@ -42,23 +42,50 @@ while $i < $tbNum sql insert into $tb values ($ms , $x , $x , $x ) $x = $x + 1 endw + + $cc = $x * 60000 + $ms = 1601481600000 + $cc + sql insert into $tb values ($ms , NULL , NULL , NULL ) $i = $i + 1 endw -print =============== step3: avg +print =============== step3: tb sql select avg(tbcol) from tb1 sql select avg(tbcol) from tb1 where ts <= 1601481840000 sql select avg(tbcol) as b from tb1 sql select avg(tbcol) as b from tb1 interval(1d) -sql select avg(tbcol) as b from tb1 where ts <= 1601481840000s interval(1m) +sql select avg(tbcol) as b from tb1 where ts <= 1601481840000 interval(1m) +sql select bottom(tbcol, 2) from tb1 where ts <= 1601481840000 +sql select top(tbcol, 2) from tb1 where ts <= 1601481840000 +sql select percentile(tbcol, 2) from tb1 where ts <= 1601481840000 +sql select leastsquares(tbcol, 1, 1) as b from tb1 where ts <= 1601481840000 +sql show table distributed tb1 +sql select count(tbcol) as b from tb1 where ts <= 1601481840000 interval(1m) +sql select diff(tbcol) from tb1 where ts <= 1601481840000 +sql select diff(tbcol) from tb1 where tbcol > 5 and tbcol < 20 +sql select first(tbcol), last(tbcol) as b from tb1 where ts <= 1601481840000 interval(1m) +sql select count(tbcol), avg(tbcol), max(tbcol), min(tbcol), sum(tbcol), stddev(tbcol) from tb1 where ts <= 1601481840000 partition by tgcol interval(1m) +#sql select count(tbcol), avg(tbcol), max(tbcol), min(tbcol), count(tbcol) from tb1 where ts <= 1601481840000 and ts >= 1601481800000 partition by tgcol interval(1m) fill(value, 0) +sql select last_row(*) from tb1 where tbcol > 5 and tbcol < 20 + +print =============== step4: stb sql select avg(tbcol) as c from stb sql select avg(tbcol) as c from stb where ts <= 1601481840000 sql select avg(tbcol) as c from stb where tgcol < 5 and ts <= 1601481840000 sql select avg(tbcol) as c from stb interval(1m) sql select avg(tbcol) as c from stb interval(1d) -sql select avg(tbcol) as b from stb where ts <= 1601481840000s interval(1m) +sql select avg(tbcol) as b from stb where ts <= 1601481840000 interval(1m) sql select avg(tbcol) as c from stb group by tgcol -sql select avg(tbcol) as b from stb where ts <= 1601481840000s partition by tgcol interval(1m) +sql select avg(tbcol) as b from stb where ts <= 1601481840000 partition by tgcol interval(1m) +sql show table distributed stb +sql select count(tbcol) as b from stb where ts <= 1601481840000 partition by tgcol interval(1m) +sql select diff(tbcol) from stb where ts <= 1601481840000 +sql select first(tbcol), last(tbcol) as c from stb group by tgcol +sql select first(tbcol), last(tbcol) as b from stb where ts <= 1601481840000 and tbcol2 is null partition by tgcol interval(1m) +sql select first(tbcol), last(tbcol) as b from stb where ts <= 1601481840000 partition by tgcol interval(1m) +sql select count(tbcol), avg(tbcol), max(tbcol), min(tbcol), sum(tbcol), stddev(tbcol) from stb where ts <= 1601481840000 partition by tgcol interval(1m) +#sql select count(tbcol), avg(tbcol), max(tbcol), min(tbcol), count(tbcol) from stb where ts <= 1601481840000 and ts >= 1601481800000 partition by tgcol interval(1m) fill(value, 0) +sql select last_row(tbcol), stddev(tbcol) from stb where tbcol > 5 and tbcol < 20 group by tgcol _OVER: system sh/exec.sh -n dnode1 -s stop -x SIGINT diff --git a/tests/script/tsim/valgrind/checkError7.sim b/tests/script/tsim/valgrind/checkError7.sim new file mode 100644 index 0000000000000000000000000000000000000000..a66ddb30df063416e0f04da0dd58de9bebed186d --- /dev/null +++ b/tests/script/tsim/valgrind/checkError7.sim @@ -0,0 +1,75 @@ +system sh/stop_dnodes.sh +system sh/deploy.sh -n dnode1 -i 1 +system sh/exec.sh -n dnode1 -s start -v +sql connect + +print ======================== create stable +sql create database d1 +sql use d1 + +$x = 0 +while $x < 128 + $tb = d1.s . $x + sql create table $tb (ts timestamp, i int) tags (j int) + $x = $x + 1 +endw + +print ======================== describe stables +# TODO : create stable error +$m = 0 +while $m < 128 + $tb = s . $m + $filter = ' . $tb + $filter = $filter . ' + sql show stables like $filter + print sql : show stables like $filter + if $rows != 1 then + print expect 1, actual: $rows + return -1 + endi + $m = $m + 1 +endw + + +print ======================== show stables + +sql show d1.stables + +print num of stables is $rows +if $rows != 128 then + return -1 +endi + +print ======================== create table + +$x = 0 +while $x < 424 + $tb = d1.t . $x + sql create table $tb using d1.s0 tags( $x ) + $x = $x + 1 +endw + +print ======================== show stables + +sql show d1.tables + +print num of tables is $rows +if $rows != 424 then + return -1 +endi + + +_OVER: +system sh/exec.sh -n dnode1 -s stop -x SIGINT +print =============== check +$null= + +system_content sh/checkValgrind.sh -n dnode1 +print cmd return result ----> [ $system_content ] +if $system_content > 2 then + return -1 +endi + +if $system_content == $null then + return -1 +endi diff --git a/tests/system-test/2-query/diff.py b/tests/system-test/2-query/diff.py index 30b588fa97025f4ee50f7f0e8bf2f07331e4711c..c6800d9a8a8ec4137551adcd2f79a4d1c8e58496 100644 --- a/tests/system-test/2-query/diff.py +++ b/tests/system-test/2-query/diff.py @@ -95,7 +95,6 @@ class TDTestCase: tdSql.error("select diff(col12) from stb_1") tdSql.error("select diff(col13) from stb_1") tdSql.error("select diff(col14) from stb_1") - tdSql.error("select ts,diff(col1),ts from stb_1") tdSql.query("select diff(col1) from stb_1") tdSql.checkRows(10) @@ -115,6 +114,79 @@ class TDTestCase: tdSql.query("select diff(col6) from stb_1") tdSql.checkRows(10) + # check selectivity + tdSql.query("select ts, diff(col1), col2 from stb_1") + tdSql.checkRows(10) + tdSql.checkData(0, 0, "2018-09-17 09:00:00.000") + tdSql.checkData(1, 0, "2018-09-17 09:00:00.001") + tdSql.checkData(2, 0, "2018-09-17 09:00:00.002") + tdSql.checkData(3, 0, "2018-09-17 09:00:00.003") + tdSql.checkData(4, 0, "2018-09-17 09:00:00.004") + tdSql.checkData(5, 0, "2018-09-17 09:00:00.005") + tdSql.checkData(6, 0, "2018-09-17 09:00:00.006") + tdSql.checkData(7, 0, "2018-09-17 09:00:00.007") + tdSql.checkData(8, 0, "2018-09-17 09:00:00.008") + tdSql.checkData(9, 0, "2018-09-17 09:00:00.009") + + tdSql.checkData(0, 1, 1) + tdSql.checkData(1, 1, 1) + tdSql.checkData(2, 1, 1) + tdSql.checkData(3, 1, 1) + tdSql.checkData(4, 1, 1) + tdSql.checkData(5, 1, 1) + tdSql.checkData(6, 1, 1) + tdSql.checkData(7, 1, 1) + tdSql.checkData(8, 1, 1) + tdSql.checkData(9, 1, 1) + + tdSql.checkData(0, 2, 0) + tdSql.checkData(1, 2, 1) + tdSql.checkData(2, 2, 2) + tdSql.checkData(3, 2, 3) + tdSql.checkData(4, 2, 4) + tdSql.checkData(5, 2, 5) + tdSql.checkData(6, 2, 6) + tdSql.checkData(7, 2, 7) + tdSql.checkData(8, 2, 8) + tdSql.checkData(9, 2, 9) + + tdSql.query("select ts, diff(col1), col2 from stb order by ts") + tdSql.checkRows(10) + + tdSql.checkData(0, 0, "2018-09-17 09:00:00.000") + tdSql.checkData(1, 0, "2018-09-17 09:00:00.001") + tdSql.checkData(2, 0, "2018-09-17 09:00:00.002") + tdSql.checkData(3, 0, "2018-09-17 09:00:00.003") + tdSql.checkData(4, 0, "2018-09-17 09:00:00.004") + tdSql.checkData(5, 0, "2018-09-17 09:00:00.005") + tdSql.checkData(6, 0, "2018-09-17 09:00:00.006") + tdSql.checkData(7, 0, "2018-09-17 09:00:00.007") + tdSql.checkData(8, 0, "2018-09-17 09:00:00.008") + tdSql.checkData(9, 0, "2018-09-17 09:00:00.009") + + tdSql.checkData(0, 1, 1) + tdSql.checkData(1, 1, 1) + tdSql.checkData(2, 1, 1) + tdSql.checkData(3, 1, 1) + tdSql.checkData(4, 1, 1) + tdSql.checkData(5, 1, 1) + tdSql.checkData(6, 1, 1) + tdSql.checkData(7, 1, 1) + tdSql.checkData(8, 1, 1) + tdSql.checkData(9, 1, 1) + + tdSql.checkData(0, 2, 0) + tdSql.checkData(1, 2, 1) + tdSql.checkData(2, 2, 2) + tdSql.checkData(3, 2, 3) + tdSql.checkData(4, 2, 4) + tdSql.checkData(5, 2, 5) + tdSql.checkData(6, 2, 6) + tdSql.checkData(7, 2, 7) + tdSql.checkData(8, 2, 8) + tdSql.checkData(9, 2, 9) + + tdSql.execute('''create table stb1(ts timestamp, col1 tinyint, col2 smallint, col3 int, col4 bigint, col5 float, col6 double, col7 bool, col8 binary(20), col9 nchar(20), col11 tinyint unsigned, col12 smallint unsigned, col13 int unsigned, col14 bigint unsigned) tags(loc nchar(20))''') tdSql.execute("create table stb1_1 using stb tags('shanghai')") diff --git a/tests/system-test/2-query/function_diff.py b/tests/system-test/2-query/function_diff.py index 5e95510c1d7337cfe81850249a375d35ba307c7c..99e87e6cd647a9731052014cc848d890d81c9362 100644 --- a/tests/system-test/2-query/function_diff.py +++ b/tests/system-test/2-query/function_diff.py @@ -283,14 +283,14 @@ class TDTestCase: tdSql.error(self.diff_query_form(alias=", diff(c1)")) # mix with calculation function 2 # tdSql.error(self.diff_query_form(alias=" + 2")) # mix with arithmetic 1 tdSql.error(self.diff_query_form(alias=" + avg(c1)")) # mix with arithmetic 2 - tdSql.error(self.diff_query_form(alias=", c2")) # mix with other 1 + tdSql.query(self.diff_query_form(alias=", c2")) # mix with other 1 # tdSql.error(self.diff_query_form(table_expr="stb1")) # select stb directly stb_join = { "col": "stb1.c1", "table_expr": "stb1, stb2", "condition": "where stb1.ts=stb2.ts and stb1.st1=stb2.st2 order by stb1.ts" } - tdSql.error(self.diff_query_form(**stb_join)) # stb join + tdSql.query(self.diff_query_form(**stb_join)) # stb join interval_sql = { "condition": "where ts>0 and ts < now interval(1h) fill(next)" } diff --git a/tests/system-test/2-query/max_partition.py b/tests/system-test/2-query/max_partition.py index 109c9075f54d60ee2071a889975e322fde37d09f..4b9996d9c3b1d45f52e184f1da4ec8e59714feaa 100644 --- a/tests/system-test/2-query/max_partition.py +++ b/tests/system-test/2-query/max_partition.py @@ -193,20 +193,11 @@ class TDTestCase: tdSql.query("select c1 , DERIVATIVE(c1,2,1) from stb partition by c1 order by c1") tdSql.checkRows(90) # bug need fix - # tdSql.checkData(0,1,None) - - - - - - + tdSql.checkData(0,1,None) - # bug need fix - # tdSql.query(" select tbname , max(c1) from stb partition by tbname order by tbname slimit 5 soffset 0 ") - # tdSql.checkRows(5) - # tdSql.query(" select tbname , max(c1) from stb partition by tbname order by tbname slimit 5 soffset 1 ") - # tdSql.checkRows(5) + tdSql.query(" select tbname , max(c1) from stb partition by tbname order by tbname slimit 5 soffset 0 ") + tdSql.checkRows(10) tdSql.query(" select tbname , max(c1) from sub_stb_1 partition by tbname interval(10s) sliding(5s) ") diff --git a/tests/system-test/7-tmq/TD-17803.py b/tests/system-test/7-tmq/TD-17803.py deleted file mode 100644 index 771ff83a29a02867c319a4b7ee6267b24eba24a5..0000000000000000000000000000000000000000 --- a/tests/system-test/7-tmq/TD-17803.py +++ /dev/null @@ -1,198 +0,0 @@ -from distutils.log import error -import taos -import sys -import time -import socket -import os -import threading -import subprocess -import platform - -from util.log import * -from util.sql import * -from util.cases import * -from util.dnodes import * -from util.common import * -sys.path.append("./7-tmq") -from tmqCommon import * - - - -class TDTestCase: - def __init__(self): - self.snapshot = 0 - self.replica = 3 - self.vgroups = 3 - self.ctbNum = 2 - self.rowsPerTbl = 2 - - def init(self, conn, logSql): - tdLog.debug(f"start to excute {__file__}") - tdSql.init(conn.cursor()) - #tdSql.init(conn.cursor(), logSql) # output sql.txt file - - def checkFileContent(self, consumerId, queryString): - buildPath = tdCom.getBuildPath() - cfgPath = tdCom.getClientCfgPath() - dstFile = '%s/../log/dstrows_%d.txt'%(cfgPath, consumerId) - cmdStr = '%s/build/bin/taos -c %s -s "%s >> %s"'%(buildPath, cfgPath, queryString, dstFile) - tdLog.info(cmdStr) - os.system(cmdStr) - - consumeRowsFile = '%s/../log/consumerid_%d.txt'%(cfgPath, consumerId) - tdLog.info("rows file: %s, %s"%(consumeRowsFile, dstFile)) - - consumeFile = open(consumeRowsFile, mode='r') - queryFile = open(dstFile, mode='r') - - # skip first line for it is schema - queryFile.readline() - - while True: - dst = queryFile.readline() - src = consumeFile.readline() - - if dst: - if dst != src: - tdLog.exit("consumerId %d consume rows is not match the rows by direct query"%consumerId) - else: - break - return - - def prepareTestEnv(self): - tdLog.printNoPrefix("======== prepare test env include database, stable, ctables, and insert data: ") - paraDict = {'dbName': 'dbt', - 'dropFlag': 1, - 'event': '', - 'vgroups': 4, - 'stbName': 'stb', - 'colPrefix': 'c', - 'tagPrefix': 't', - 'colSchema': [{'type': 'INT', 'count':1},{'type': 'BIGINT', 'count':1}], - 'tagSchema': [{'type': 'INT', 'count':1},{'type': 'BIGINT', 'count':1}], - 'ctbPrefix': 'ctb', - 'ctbStartIdx': 0, - 'ctbNum': 2, - 'rowsPerTbl': 1000, - 'batchNum': 10, - 'startTs': 1640966400000, # 2022-01-01 00:00:00.000 - 'pollDelay': 3, - 'showMsg': 1, - 'showRow': 1, - 'snapshot': 0} - - paraDict['vgroups'] = self.vgroups - paraDict['ctbNum'] = self.ctbNum - paraDict['rowsPerTbl'] = self.rowsPerTbl - - tmqCom.initConsumerTable() - tdCom.create_database(tdSql, paraDict["dbName"],paraDict["dropFlag"], vgroups=paraDict["vgroups"],replica=self.replica) - tdLog.info("create stb") - tmqCom.create_stable(tdSql, dbName=paraDict["dbName"],stbName=paraDict["stbName"]) - tdLog.info("create ctb") - tmqCom.create_ctable(tdSql, dbName=paraDict["dbName"],stbName=paraDict["stbName"],ctbPrefix=paraDict['ctbPrefix'], - ctbNum=paraDict["ctbNum"],ctbStartIdx=paraDict['ctbStartIdx']) - tdLog.info("insert data") - tmqCom.insert_data_interlaceByMultiTbl(tsql=tdSql,dbName=paraDict["dbName"],ctbPrefix=paraDict["ctbPrefix"], - ctbNum=paraDict["ctbNum"],rowsPerTbl=paraDict["rowsPerTbl"],batchNum=paraDict["batchNum"], - startTs=paraDict["startTs"],ctbStartIdx=paraDict['ctbStartIdx']) - # tmqCom.insert_data_with_autoCreateTbl(tsql=tdSql,dbName=paraDict["dbName"],stbName=paraDict["stbName"],ctbPrefix="ctbx", - # ctbNum=paraDict["ctbNum"],rowsPerTbl=paraDict["rowsPerTbl"],batchNum=paraDict["batchNum"], - # startTs=paraDict["startTs"],ctbStartIdx=paraDict['ctbStartIdx']) - # tmqCom.asyncInsertDataByInterlace(paraDict) - tdLog.printNoPrefix("11111111111111111111111") - tmqCom.create_ntable(tdSql, dbname=paraDict["dbName"], tbname_prefix="ntb", tbname_index_start_num = 1, column_elm_list=paraDict["colSchema"], colPrefix='c', tblNum=1) - tdLog.printNoPrefix("222222222222222") - tmqCom.insert_rows_into_ntbl(tdSql, dbname=paraDict["dbName"], tbname_prefix="ntb", tbname_index_start_num = 1, column_ele_list=paraDict["colSchema"], startTs=paraDict["startTs"], tblNum=1, rows=2) # tdLog.info("restart taosd to ensure that the data falls into the disk") - - tdLog.printNoPrefix("333333333333333333333") - tdSql.query("drop database %s"%paraDict["dbName"]) - tdLog.printNoPrefix("44444444444444444") - return - - def tmqCase1(self): - tdLog.printNoPrefix("======== test case 1: ") - - # create and start thread - paraDict = {'dbName': 'dbt', - 'dropFlag': 1, - 'event': '', - 'vgroups': 4, - 'stbName': 'stb', - 'colPrefix': 'c', - 'tagPrefix': 't', - 'colSchema': [{'type': 'INT', 'count':1},{'type': 'BIGINT', 'count':1},{'type': 'DOUBLE', 'count':1},{'type': 'BINARY', 'len':32, 'count':1},{'type': 'NCHAR', 'len':32, 'count':1},{'type': 'TIMESTAMP', 'count':1}], - 'tagSchema': [{'type': 'INT', 'count':1},{'type': 'BIGINT', 'count':1},{'type': 'DOUBLE', 'count':1},{'type': 'BINARY', 'len':32, 'count':1},{'type': 'NCHAR', 'len':32, 'count':1}], - 'ctbPrefix': 'ctb', - 'ctbStartIdx': 0, - 'ctbNum': 100, - 'rowsPerTbl': 1000, - 'batchNum': 100, - 'startTs': 1640966400000, # 2022-01-01 00:00:00.000 - 'pollDelay': 3, - 'showMsg': 1, - 'showRow': 1, - 'snapshot': 1} - - paraDict['vgroups'] = self.vgroups - paraDict['ctbNum'] = self.ctbNum - paraDict['rowsPerTbl'] = self.rowsPerTbl - - tdLog.info("create topics from stb1") - topicFromStb1 = 'topic_stb1' - queryString = "select ts, c1, c2 from %s.%s where t4 == 'beijing' or t4 == 'changsha' "%(paraDict['dbName'], paraDict['stbName']) - sqlString = "create topic %s as %s" %(topicFromStb1, queryString) - tdLog.info("create topic sql: %s"%sqlString) - tdSql.execute(sqlString) - - consumerId = 0 - expectrowcnt = paraDict["rowsPerTbl"] * paraDict["ctbNum"] - topicList = topicFromStb1 - ifcheckdata = 0 - ifManualCommit = 0 - keyList = 'group.id:cgrp1,\ - enable.auto.commit:false,\ - auto.commit.interval.ms:6000,\ - auto.offset.reset:earliest' - tmqCom.insertConsumerInfo(consumerId, expectrowcnt,topicList,keyList,ifcheckdata,ifManualCommit) - - tdLog.info("start consume processor") - pollDelay = 100 - showMsg = 1 - showRow = 1 - tmqCom.startTmqSimProcess(pollDelay=paraDict['pollDelay'],dbName=paraDict["dbName"],showMsg=paraDict['showMsg'], showRow=paraDict['showRow'],snapshot=paraDict['snapshot']) - - tdLog.info("start to check consume result") - expectRows = 1 - resultList = tmqCom.selectConsumeResult(expectRows) - totalConsumeRows = 0 - for i in range(expectRows): - totalConsumeRows += resultList[i] - - tdSql.query(queryString) - totalRowsInserted = tdSql.getRows() - - tdLog.info("act consume rows: %d, act insert rows: %d, expect consume rows: %d, "%(totalConsumeRows, totalRowsInserted, expectrowcnt)) - - if totalConsumeRows != expectrowcnt: - tdLog.exit("tmq consume rows error!") - - # tmqCom.checkFileContent(consumerId, queryString) - - tmqCom.waitSubscriptionExit(tdSql, topicFromStb1) - tdSql.query("drop topic %s"%topicFromStb1) - - tdLog.printNoPrefix("======== test case 1 end ...... ") - - def run(self): - self.prepareTestEnv() - # self.tmqCase1() - - def stop(self): - tdSql.close() - tdLog.success(f"{__file__} successfully executed") - -event = threading.Event() - -tdCases.addLinux(__file__, TDTestCase()) -tdCases.addWindows(__file__, TDTestCase()) diff --git a/tests/system-test/7-tmq/db.py b/tests/system-test/7-tmq/db.py index fd793fd841f76a402ba35fed68013167133d9a62..1fd0638d172fe6dfb6f19c1617e1eebc38ccac41 100644 --- a/tests/system-test/7-tmq/db.py +++ b/tests/system-test/7-tmq/db.py @@ -118,7 +118,7 @@ class TDTestCase: if dropFlag == 1: tsql.execute("drop database if exists %s"%(dbName)) - tsql.execute("create database if not exists %s vgroups %d replica %d"%(dbName, vgroups, replica)) + tsql.execute("create database if not exists %s vgroups %d replica %d wal_retention_period -1 wal_retention_size -1"%(dbName, vgroups, replica)) tdLog.debug("complete to create database %s"%(dbName)) return diff --git a/tests/system-test/7-tmq/tmqDelete-1ctb.py b/tests/system-test/7-tmq/tmqDelete-1ctb.py index a2a429771cc4fddcbf8ba86c64e0d5cfc90ff7cd..bedb36e505fa0f0d136d33aeeb472287d7566c54 100644 --- a/tests/system-test/7-tmq/tmqDelete-1ctb.py +++ b/tests/system-test/7-tmq/tmqDelete-1ctb.py @@ -52,7 +52,7 @@ class TDTestCase: paraDict['rowsPerTbl'] = self.rowsPerTbl tmqCom.initConsumerTable() - tdCom.create_database(tdSql, paraDict["dbName"],paraDict["dropFlag"], vgroups=paraDict["vgroups"],replica=1) + tdCom.create_database(tdSql, paraDict["dbName"],paraDict["dropFlag"], vgroups=paraDict["vgroups"],replica=1,wal_retention_size=-1, wal_retention_period=-1) tdLog.info("create stb") tmqCom.create_stable(tdSql, dbName=paraDict["dbName"],stbName=paraDict["stbName"]) tdLog.info("create ctb") diff --git a/tests/system-test/7-tmq/tmqDelete-multiCtb.py b/tests/system-test/7-tmq/tmqDelete-multiCtb.py index fa32efbd0b6b44d8077b182b5dd6c7dbe81aab2c..94ca16bc6f4b6abf260f291380e1c1f0079d5fa1 100644 --- a/tests/system-test/7-tmq/tmqDelete-multiCtb.py +++ b/tests/system-test/7-tmq/tmqDelete-multiCtb.py @@ -52,7 +52,7 @@ class TDTestCase: paraDict['rowsPerTbl'] = self.rowsPerTbl tmqCom.initConsumerTable() - tdCom.create_database(tdSql, paraDict["dbName"],paraDict["dropFlag"], vgroups=paraDict["vgroups"],replica=1) + tdCom.create_database(tdSql, paraDict["dbName"],paraDict["dropFlag"], vgroups=paraDict["vgroups"],replica=1,wal_retention_size=-1, wal_retention_period=-1) tdLog.info("create stb") tmqCom.create_stable(tdSql, dbName=paraDict["dbName"],stbName=paraDict["stbName"]) tdLog.info("create ctb") diff --git a/tests/system-test/7-tmq/tmqDnodeRestart.py b/tests/system-test/7-tmq/tmqDnodeRestart.py index 5117ee3d24c289e695eb3113e643e94a6eb01e53..9a11106e3ea464415815dc020870f60cbaf21fb7 100644 --- a/tests/system-test/7-tmq/tmqDnodeRestart.py +++ b/tests/system-test/7-tmq/tmqDnodeRestart.py @@ -53,7 +53,7 @@ class TDTestCase: paraDict['rowsPerTbl'] = self.rowsPerTbl tmqCom.initConsumerTable() - tdCom.create_database(tdSql, paraDict["dbName"],paraDict["dropFlag"], vgroups=paraDict["vgroups"],replica=1) + tdCom.create_database(tdSql, paraDict["dbName"],paraDict["dropFlag"], vgroups=paraDict["vgroups"],replica=1,wal_retention_size=-1, wal_retention_period=-1) tdLog.info("create stb") tmqCom.create_stable(tdSql, dbName=paraDict["dbName"],stbName=paraDict["stbName"]) tdLog.info("create ctb") diff --git a/tests/system-test/7-tmq/tmqUpdateWithConsume.py b/tests/system-test/7-tmq/tmqUpdateWithConsume.py index 4f21beffc40e825ce143bf23060f566048ec16b4..2dd3a061c62bf5ac63a135e99424ac549ef83a54 100644 --- a/tests/system-test/7-tmq/tmqUpdateWithConsume.py +++ b/tests/system-test/7-tmq/tmqUpdateWithConsume.py @@ -52,7 +52,7 @@ class TDTestCase: paraDict['rowsPerTbl'] = self.rowsPerTbl tmqCom.initConsumerTable() - tdCom.create_database(tdSql, paraDict["dbName"],paraDict["dropFlag"], vgroups=paraDict["vgroups"],replica=1) + tdCom.create_database(tdSql, paraDict["dbName"],paraDict["dropFlag"], vgroups=paraDict["vgroups"],replica=1, wal_retention_size=-1, wal_retention_period=-1) tdLog.info("create stb") tmqCom.create_stable(tdSql, dbName=paraDict["dbName"],stbName=paraDict["stbName"]) tdLog.info("create ctb")