提交 cbcb1464 编写于 作者: G Ganlin Zhao

Merge branch '3.0' into fix/TD-18457

......@@ -104,15 +104,15 @@ Each row contains the device ID, time stamp, collected metrics (current, voltage
## Metric
Metric refers to the physical quantity collected by sensors, equipment or other types of data collection devices, such as current, voltage, temperature, pressure, GPS position, etc., which change with time, and the data type can be integer, float, Boolean, or strings. As time goes by, the amount of collected metric data stored increases.
Metric refers to the physical quantity collected by sensors, equipment or other types of data collection devices, such as current, voltage, temperature, pressure, GPS position, etc., which change with time, and the data type can be integer, float, Boolean, or strings. As time goes by, the amount of collected metric data stored increases. In the smart meters example, current, voltage and phase are the metrics.
## Label/Tag
Label/Tag refers to the static properties of sensors, equipment or other types of data collection devices, which do not change with time, such as device model, color, fixed location of the device, etc. The data type can be any type. Although static, TDengine allows users to add, delete or update tag values at any time. Unlike the collected metric data, the amount of tag data stored does not change over time.
Label/Tag refers to the static properties of sensors, equipment or other types of data collection devices, which do not change with time, such as device model, color, fixed location of the device, etc. The data type can be any type. Although static, TDengine allows users to add, delete or update tag values at any time. Unlike the collected metric data, the amount of tag data stored does not change over time. In the meters example, `location` and `groupid` are the tags.
## Data Collection Point
Data Collection Point (DCP) refers to hardware or software that collects metrics based on preset time periods or triggered by events. A data collection point can collect one or multiple metrics, but these metrics are collected at the same time and have the same time stamp. For some complex equipment, there are often multiple data collection points, and the sampling rate of each collection point may be different, and fully independent. For example, for a car, there could be a data collection point to collect GPS position metrics, a data collection point to collect engine status metrics, and a data collection point to collect the environment metrics inside the car. So in this example the car would have three data collection points.
Data Collection Point (DCP) refers to hardware or software that collects metrics based on preset time periods or triggered by events. A data collection point can collect one or multiple metrics, but these metrics are collected at the same time and have the same time stamp. For some complex equipment, there are often multiple data collection points, and the sampling rate of each collection point may be different, and fully independent. For example, for a car, there could be a data collection point to collect GPS position metrics, a data collection point to collect engine status metrics, and a data collection point to collect the environment metrics inside the car. So in this example the car would have three data collection points. In the smart meters example, d1001, d1002, d1003, and d1004 are the data collection points.
## Table
......@@ -137,7 +137,7 @@ The design of one table for one data collection point will require a huge number
STable is a template for a type of data collection point. A STable contains a set of data collection points (tables) that have the same schema or data structure, but with different static attributes (tags). To describe a STable, in addition to defining the table structure of the metrics, it is also necessary to define the schema of its tags. The data type of tags can be int, float, string, and there can be multiple tags, which can be added, deleted, or modified afterward. If the whole system has N different types of data collection points, N STables need to be established.
In the design of TDengine, **a table is used to represent a specific data collection point, and STable is used to represent a set of data collection points of the same type**.
In the design of TDengine, **a table is used to represent a specific data collection point, and STable is used to represent a set of data collection points of the same type**. In the smart meters example, we can create a super table named `meters`.
## Subtable
......@@ -156,9 +156,9 @@ The relationship between a STable and the subtables created based on this STable
Queries can be executed on both a table (subtable) and a STable. For a query on a STable, TDengine will treat the data in all its subtables as a whole data set for processing. TDengine will first find the subtables that meet the tag filter conditions, then scan the time-series data of these subtables to perform aggregation operation, which reduces the number of data sets to be scanned which in turn greatly improves the performance of data aggregation across multiple DCPs. In essence, querying a supertable is a very efficient aggregate query on multiple DCPs of the same type.
In TDengine, it is recommended to use a subtable instead of a regular table for a DCP. In the meters example, we can create subtables like d1001, d1002, d1003, and d1004 under super table meters.
In TDengine, it is recommended to use a subtable instead of a regular table for a DCP. In the smart meters example, we can create subtables like d1001, d1002, d1003, and d1004 under super table meters.
To better understand the data model using super table and subtable, please refer to the diagram below which demonstrates the data model of meters example. ![Meters Data Model Diagram](./supertable.webp)
To better understand the data model using metri, tags, super table and subtable, please refer to the diagram below which demonstrates the data model of the smart meters example. ![Meters Data Model Diagram](./supertable.webp)
## Database
......
......@@ -11,7 +11,7 @@ When using TDengine to store and query data, the most important part of the data
- The format must be `YYYY-MM-DD HH:mm:ss.MS`, the default time precision is millisecond (ms), for example `2017-08-12 18:25:58.128`
- Internal function `now` can be used to get the current timestamp on the client side
- The current timestamp of the client side is applied when `now` is used to insert data
- Epoch Time:timestamp can also be a long integer number, which means the number of seconds, milliseconds or nanoseconds, depending on the time precision, from 1970-01-01 00:00:00.000 (UTC/GMT)
- Epoch Time:timestamp can also be a long integer number, which means the number of seconds, milliseconds or nanoseconds, depending on the time precision, from UTC 1970-01-01 00:00:00.
- Add/subtract operations can be carried out on timestamps. For example `now-2h` means 2 hours prior to the time at which query is executed. The units of time in operations can be b(nanosecond), u(microsecond), a(millisecond), s(second), m(minute), h(hour), d(day), or w(week). So `select * from t1 where ts > now-2w and ts <= now-1w` means the data between two weeks ago and one week ago. The time unit can also be n (calendar month) or y (calendar year) when specifying the time window for down sampling operations.
Time precision in TDengine can be set by the `PRECISION` parameter when executing `CREATE DATABASE`. The default time precision is millisecond. In the statement below, the precision is set to nanonseconds.
......
---
title: 产品简介
description: 简要介绍 TDengine 的主要功能
toc_max_heading_level: 2
---
......
---
sidebar_label: 基本概念
title: 数据模型和基本概念
description: TDengine 的数据模型和基本概念
---
为了便于解释基本概念,便于撰写示例程序,整个 TDengine 文档以智能电表作为典型时序数据场景。假设每个智能电表采集电流、电压、相位三个量,有多个智能电表,每个电表有位置 location 和分组 group ID 的静态属性. 其采集的数据类似如下的表格:
......
---
sidebar_label: Docker
title: 通过 Docker 快速体验 TDengine
description: 使用 Docker 快速体验 TDengine 的高效写入和查询
---
本节首先介绍如何通过 Docker 快速体验 TDengine,然后介绍如何在 Docker 环境下体验 TDengine 的写入和查询功能。如果你不熟悉 Docker,请使用[安装包的方式快速体验](../../get-started/package/)。如果您希望为 TDengine 贡献代码或对内部技术实现感兴趣,请参考 [TDengine GitHub 主页](https://github.com/taosdata/TDengine) 下载源码构建和安装.
......
---
sidebar_label: 安装包
title: 使用安装包立即开始
description: 使用安装包快速体验 TDengine
---
import Tabs from "@theme/Tabs";
......
---
title: 建立连接
description: "本节介绍如何使用连接器建立与 TDengine 的连接,给出连接器安装、连接的简单说明。"
description: 使用连接器建立与 TDengine 的连接,以及连接器的安装和连接
---
import Tabs from "@theme/Tabs";
......
---
sidebar_label: 数据建模
title: TDengine 数据建模
description: TDengine 中如何建立数据模型
---
TDengine 采用类关系型数据模型,需要建库、建表。因此对于一个具体的应用场景,需要考虑库、超级表和普通表的设计。本节不讨论细致的语法规则,只介绍概念。
......
---
sidebar_label: 写入数据
title: 写入数据
description: TDengine 的各种写入方式
---
TDengine 支持多种写入协议,包括 SQL,InfluxDB Line 协议, OpenTSDB Telnet 协议,OpenTSDB JSON 格式协议。数据可以单条插入,也可以批量插入,可以插入一个数据采集点的数据,也可以同时插入多个数据采集点的数据。同时,TDengine 支持多线程插入,支持时间乱序数据插入,也支持历史数据插入。InfluxDB Line 协议、OpenTSDB Telnet 协议和 OpenTSDB JSON 格式协议是 TDengine 支持的三种无模式写入协议。使用无模式方式写入无需提前创建超级表和子表,并且引擎能自适用数据对表结构做调整。
......
---
sidebar_label: 查询数据
title: 查询数据
description: "主要查询功能,通过连接器执行同步查询和异步查询"
---
......
---
title: 开发指南
sidebar_label: 开发指南
description: 让开发者能够快速上手的指南
---
开发一个应用,如果你准备采用TDengine作为时序数据处理的工具,那么有如下几个事情要做:
......
---
sidebar_label: 错误码
title: TDengine C/C++ 连接器错误码
description: C/C++ 连接器的错误码列表和详细说明
---
本文中详细列举了在使用 TDengine C/C++ 连接器时客户端可能得到的错误码以及所要采取的相应动作。其它语言的连接器在使用原生连接方式时也会所得到的返回码返回给连接器的调用者。
......
---
sidebar_label: 手动部署
title: 集群部署和管理
description: 使用命令行工具手动部署 TDengine 集群
---
## 准备工作
......
---
sidebar_label: Kubernetes
title: 在 Kubernetes 上部署 TDengine 集群
description: 利用 Kubernetes 部署 TDengine 集群的详细指南
---
作为面向云原生架构设计的时序数据库,TDengine 支持 Kubernetes 部署。这里介绍如何使用 YAML 文件一步一步从头创建一个 TDengine 集群,并重点介绍 Kubernetes 环境下 TDengine 的常用操作。
......
---
sidebar_label: Helm
title: 使用 Helm 部署 TDengine 集群
description: 使用 Helm 部署 TDengine 集群的详细指南
---
Helm 是 Kubernetes 的包管理器,上一节使用 Kubernets 部署 TDengine 集群的操作已经足够简单,但 Helm 依然可以提供更强大的能力。
......
---
sidebar_label: 部署集群
title: 部署集群
description: 部署 TDengine 集群的多种方式
---
TDengine 支持集群,提供水平扩展的能力。如果需要获得更高的处理能力,只需要多增加节点即可。TDengine 采用虚拟节点技术,将一个节点虚拟化为多个虚拟节点,以实现负载均衡。同时,TDengine可以将多个节点上的虚拟节点组成虚拟节点组,通过多副本机制,以保证供系统的高可用。TDengine的集群功能完全开源。
......
......@@ -11,7 +11,7 @@ description: "TDengine 支持的数据类型: 时间戳、浮点型、JSON 类
- 时间格式为 `YYYY-MM-DD HH:mm:ss.MS`,默认时间分辨率为毫秒。比如:`2017-08-12 18:25:58.128`
- 内部函数 now 是客户端的当前时间
- 插入记录时,如果时间戳为 now,插入数据时使用提交这条记录的客户端的当前时间
- Epoch Time:时间戳也可以是一个长整数,表示从格林威治时间 1970-01-01 00:00:00.000 (UTC/GMT) 开始的毫秒数(相应地,如果所在 Database 的时间精度设置为“微秒”,则长整型格式的时间戳含义也就对应于从格林威治时间 1970-01-01 00:00:00.000 (UTC/GMT) 开始的微秒数;纳秒精度逻辑类似。)
- Epoch Time:时间戳也可以是一个长整数,表示从 UTC 时间 1970-01-01 00:00:00 开始的毫秒数。相应地,如果所在 Database 的时间精度设置为“微秒”,则长整型格式的时间戳含义也就对应于从 UTC 时间 1970-01-01 00:00:00 开始的微秒数;纳秒精度逻辑类似。
- 时间可以加减,比如 now-2h,表明查询时刻向前推 2 个小时(最近 2 小时)。数字后面的时间单位可以是 b(纳秒)、u(微秒)、a(毫秒)、s(秒)、m(分)、h(小时)、d(天)、w(周)。 比如 `select * from t1 where ts > now-2w and ts <= now-1w`,表示查询两周前整整一周的数据。在指定降采样操作(down sampling)的时间窗口(interval)时,时间单位还可以使用 n (自然月) 和 y (自然年)。
TDengine 缺省的时间戳精度是毫秒,但通过在 `CREATE DATABASE` 时传递的 PRECISION 参数也可以支持微秒和纳秒。
......
---
title: 表管理
sidebar_label:
description: 对表的各种管理操作
---
## 创建表
......
---
sidebar_label: 超级表管理
title: 超级表 STable 管理
description: 对超级表的各种管理操作
---
## 创建超级表
......
---
sidebar_label: 数据写入
title: 数据写入
description: 写入数据的详细语法
---
## 写入语法
......
---
sidebar_label: 数据查询
title: 数据查询
description: 查询数据的详细语法
---
## 查询语法
......
---
sidebar_label: 函数
title: 函数
description: TDengine 支持的函数列表
toc_max_heading_level: 4
---
......
---
sidebar_label: 时序数据特色查询
title: 时序数据特色查询
description: TDengine 提供的时序数据特有的查询功能
---
TDengine 是专为时序数据而研发的大数据平台,存储和计算都针对时序数据的特定进行了量身定制,在支持标准 SQL 的基础之上,还提供了一系列贴合时序业务场景的特色查询语法,极大的方便时序场景的应用开发。
......
---
sidebar_label: 数据订阅
title: 数据订阅
description: TDengine 消息队列提供的数据订阅功能
---
TDengine 3.0.0.0 开始对消息队列做了大幅的优化和增强以简化用户的解决方案。
......
---
sidebar_label: 流式计算
title: 流式计算
description: 流式计算的相关 SQL 的详细语法
---
......
---
sidebar_label: 运算符
title: 运算符
description: TDengine 支持的所有运算符
---
## 算术运算符
......
---
sidebar_label: JSON 类型使用说明
title: JSON 类型使用说明
description: 对 JSON 类型如何使用的详细说明
---
......
---
title: 转义字符说明
sidebar_label: 转义字符
description: TDengine 中使用转义字符的详细规则
---
## 转义字符表
......
---
sidebar_label: 命名与边界限制
title: 命名与边界限制
description: 合法字符集和命名中的限制规则
---
## 名称命名规则
......
---
sidebar_label: 保留关键字
title: TDengine 保留关键字
description: TDengine 保留关键字的详细列表
---
## 保留关键字
......
---
sidebar_label: 集群管理
title: 集群管理
description: 管理集群的 SQL 命令的详细解析
---
组成 TDengine 集群的物理实体是 dnode (data node 的缩写),它是一个运行在操作系统之上的进程。在 dnode 中可以建立负责时序数据存储的 vnode (virtual node),在多节点集群环境下当某个数据库的 replica 为 3 时,该数据库中的每个 vgroup 由 3 个 vnode 组成;当数据库的 replica 为 1 时,该数据库中的每个 vgroup 由 1 个 vnode 组成。如果要想配置某个数据库为多副本,则集群中的 dnode 数量至少为 3。在 dnode 还可以创建 mnode (management node),单个集群中最多可以创建三个 mnode。在 TDengine 3.0.0.0 中为了支持存算分离,引入了一种新的逻辑节点 qnode (query node),qnode 和 vnode 既可以共存在一个 dnode 中,也可以完全分离在不同的 dnode 上。
......
---
sidebar_label: 元数据
title: 存储元数据的 Information_Schema 数据库
description: Information_Schema 数据库中存储了系统中所有的元数据信息
---
TDengine 内置了一个名为 `INFORMATION_SCHEMA` 的数据库,提供对数据库元数据、数据库系统信息和状态的访问,例如数据库或表的名称,当前执行的 SQL 语句等。该数据库存储有关 TDengine 维护的所有其他数据库的信息。它包含多个只读表。实际上,这些表都是视图,而不是基表,因此没有与它们关联的文件。所以对这些表只能查询,不能进行 INSERT 等写入操作。`INFORMATION_SCHEMA` 数据库旨在以一种更一致的方式来提供对 TDengine 支持的各种 SHOW 语句(如 SHOW TABLES、SHOW DATABASES)所提供的信息的访问。与 SHOW 语句相比,使用 SELECT ... FROM INFORMATION_SCHEMA.tablename 具有以下优点:
......
---
sidebar_label: 统计数据
title: 存储统计数据的 Performance_Schema 数据库
description: Performance_Schema 数据库中存储了系统中的各种统计信息
---
TDengine 3.0 版本开始提供一个内置数据库 `performance_schema`,其中存储了与性能有关的统计数据。本节详细介绍其中的表和表结构。
......
---
sidebar_label: SHOW 命令
title: 使用 SHOW 命令查看系统元数据
description: SHOW 命令的完整列表
---
SHOW 命令可以用来获取简要的系统信息。若想获取系统中详细的各种元数据、系统信息和状态,请使用 select 语句查询 INFORMATION_SCHEMA 数据库中的表。
......
---
sidebar_label: 权限管理
title: 权限管理
description: 企业版中才具有的权限管理功能
---
本节讲述如何在 TDengine 中进行权限管理的相关操作。
......
---
sidebar_label: 自定义函数
title: 用户自定义函数
description: 使用 UDF 的详细指南
---
除了 TDengine 的内置函数以外,用户还可以编写自己的函数逻辑并加入TDengine系统中。
......
---
sidebar_label: 索引
title: 使用索引
description: 索引功能的使用细节
---
TDengine 从 3.0.0.0 版本开始引入了索引功能,支持 SMA 索引和 FULLTEXT 索引。
......
---
sidebar_label: 异常恢复
title: 异常恢复
description: 如何终止出现问题的连接、查询和事务以使系统恢复正常
---
在一个复杂的应用场景中,连接和查询任务等有可能进入一种错误状态或者耗时过长迟迟无法结束,此时需要有能够终止这些连接或任务的方法。
......
---
title: TDinsight - 基于Grafana的TDengine零依赖监控解决方案
title: TDinsight
sidebar_label: TDinsight
description: 基于Grafana的TDengine零依赖监控解决方案
---
TDinsight 是使用监控数据库和 [Grafana] 对 TDengine 进行监控的解决方案。
......
---
title: 参考手册
description: TDengine 中的各种组件的详细说明
---
参考手册是对 TDengine 本身、 TDengine 各语言连接器及自带的工具最详细的介绍。
......
---
sidebar_label: 容量规划
title: 容量规划
description: 如何规划一个 TDengine 集群所需的物理资源
---
使用 TDengine 来搭建一个物联网大数据平台,计算资源、存储资源需要根据业务场景进行规划。下面分别讨论系统运行所需要的内存、CPU 以及硬盘空间。
......
---
title: 容错和灾备
sidebar_label: 容错和灾备
description: TDengine 的容错和灾备功能
---
## 容错
......
---
title: 数据导入
description: 如何导入外部数据到 TDengine
---
TDengine 提供多种方便的数据导入功能,一种按脚本文件导入,一种按数据文件导入,一种是 taosdump 工具导入本身导出的文件。
......
---
title: 数据导出
description: 如何导出 TDengine 中的数据
---
为方便数据导出,TDengine 提供了两种导出方式,分别是按表导出和用 taosdump 导出。
......
---
title: 系统监控
description: 监控 TDengine 的运行状态
---
TDengine 通过 [taosKeeper](/reference/taosKeeper/) 将服务器的 CPU、内存、硬盘空间、带宽、请求数、磁盘读写速度等信息定时写入指定数据库。TDengine 还将重要的系统操作(比如登录、创建、删除数据库等)日志以及各种错误报警信息进行记录。系统管理员可以从 CLI 直接查看这个数据库,也可以在 WEB 通过图形化界面查看这些监测信息。
......
---
title: 诊断及其他
description: 一些常见问题的诊断技巧
---
## 网络连接诊断
......
---
sidebar_label: Grafana
title: Grafana
description: 使用 Grafana 与 TDengine 的详细说明
---
import Tabs from "@theme/Tabs";
......
---
sidebar_label: Prometheus
title: Prometheus
description: 使用 Prometheus 访问 TDengine
---
import Prometheus from "../14-reference/_prometheus.mdx"
......
---
sidebar_label: Telegraf
title: Telegraf 写入
description: 使用 Telegraf 向 TDengine 写入数据
---
import Telegraf from "../14-reference/_telegraf.mdx"
......
---
sidebar_label: collectd
title: collectd 写入
description: 使用 collected 向 TDengine 写入数据
---
import CollectD from "../14-reference/_collectd.mdx"
......
---
sidebar_label: StatsD
title: StatsD 直接写入
description: 使用 StatsD 向 TDengine 写入
---
import StatsD from "../14-reference/_statsd.mdx"
......
---
sidebar_label: icinga2
title: icinga2 写入
description: 使用 icinga2 写入 TDengine
---
import Icinga2 from "../14-reference/_icinga2.mdx"
......
---
sidebar_label: TCollector
title: TCollector 写入
description: 使用 TCollector 写入 TDengine
---
import TCollector from "../14-reference/_tcollector.mdx"
......
---
sidebar_label: EMQX Broker
title: EMQX Broker 写入
description: 使用 EMQX Broker 写入 TDengine
---
MQTT 是流行的物联网数据传输协议,[EMQX](https://github.com/emqx/emqx)是一开源的 MQTT Broker 软件,无需任何代码,只需要在 EMQX Dashboard 里使用“规则”做简单配置,即可将 MQTT 的数据直接写入 TDengine。EMQX 支持通过 发送到 Web 服务的方式保存数据到 TDengine,也在企业版上提供原生的 TDengine 驱动实现直接保存。
......
---
sidebar_label: HiveMQ Broker
title: HiveMQ Broker 写入
description: 使用 HivMQ Broker 写入 TDengine
---
[HiveMQ](https://www.hivemq.com/) 是一个提供免费个人版和企业版的 MQTT 代理,主要用于企业和新兴的机器到机器 M2M 通讯和内部传输,满足可伸缩性、易管理和安全特性。HiveMQ 提供了开源的插件开发包。可以通过 HiveMQ extension - TDengine 保存数据到 TDengine。详细使用方法请参考 [HiveMQ extension - TDengine 说明文档](https://github.com/huskar-t/hivemq-tdengine-extension/blob/b62a26ecc164a310104df57691691b237e091c89/README.md)
---
sidebar_label: Kafka
title: TDengine Kafka Connector 使用教程
title: TDengine Kafka Connector
description: 使用 TDengine Kafka Connector 的详细指南
---
TDengine Kafka Connector 包含两个插件: TDengine Source Connector 和 TDengine Sink Connector。用户只需提供简单的配置文件,就可以将 Kafka 中指定 topic 的数据(批量或实时)同步到 TDengine, 或将 TDengine 中指定数据库的数据(批量或实时)同步到 Kafka。
......
---
sidebar_label: 整体架构
title: 整体架构
description: TDengine 架构设计,包括:集群、存储、缓存与持久化、数据备份、多级存储等
---
## 集群与基本逻辑单元
......
---
title: 高可用
description: TDengine 的高可用设计
---
## Vnode 的高可用性
......
---
title: 负载均衡
description: TDengine 的负载均衡设计
---
TDengine 中的负载均衡主要指对时序数据的处理的负载均衡。TDengine 采用 Hash 一致性算法将一个数据库中的所有表和子表的数据均衡分散在属于该数据库的所有 vgroup 中,每张表或子表只能由一个 vgroup 处理,一个 vgroup 可能负责处理多个表或子表。
......@@ -7,7 +8,7 @@ TDengine 中的负载均衡主要指对时序数据的处理的负载均衡。TD
创建数据库时可以指定其中的 vgroup 的数量:
```sql
create database db0 vgroups 100;
create database db0 vgroups 20;
```
如何指定合适的 vgroup 的数量,这取决于系统资源。假定系统中只计划建立一个数据库,则 vgroup 数量由集群中所有 dnode 所能使用的资源决定。原则上可用的 CPU 和 Memory 越多,可建立的 vgroup 也越多。但也要考虑到磁盘性能,过多的 vgroup 在磁盘性能达到上限后反而会拖累整个系统的性能。假如系统中会建立多个数据库,则多个数据库的 vgroup 之和取决于系统中可用资源的数量。要综合考虑多个数据库之间表的数量、写入频率、数据量等多个因素在多个数据库之间分配 vgroup。实际中建议首先根据系统资源配置选择一个初始的 vgroup 数量,比如 CPU 总核数的 2 倍,以此为起点通过测试找到最佳的 vgroup 数量配置,此为系统中的 vgroup 总数。如果有多个数据库的话,再根据各个数据库的表数和数据量对 vgroup 进行分配。
......
---
title: 技术内幕
description: TDengine 的内部设计
---
```mdx-code-block
......
---
sidebar_label: TDengine + Telegraf + Grafana
title: 使用 TDengine + Telegraf + Grafana 快速搭建 IT 运维展示系统
title: TDengine + Telegraf + Grafana
description: 使用 TDengine + Telegraf + Grafana 快速搭建 IT 运维展示系统
---
## 背景介绍
......
---
sidebar_label: TDengine + collectd/StatsD + Grafana
title: 使用 TDengine + collectd/StatsD + Grafana 快速搭建 IT 运维监控系统
title: TDengine + collectd/StatsD + Grafana
description: 使用 TDengine + collectd/StatsD + Grafana 快速搭建 IT 运维监控系统
---
## 背景介绍
......
---
title: 应用实践
description: TDengine 配合其它开源组件的一些应用示例
---
```mdx-code-block
......
---
title: 常见问题及反馈
description: 一些常见问题的解决方法汇总
---
## 问题反馈
......
---
title: FAQ 及其他
description: 用户经常遇到的问题
---
```mdx-code-block
......
---
sidebar_label: TDengine 发布历史
title: TDengine 发布历史
description: TDengine 发布历史、Release Notes 及下载链接
---
import Release from "/components/ReleaseV3";
......
---
sidebar_label: taosTools 发布历史
title: taosTools 发布历史
description: taosTools 的发布历史、Release Notes 和下载链接
---
import Release from "/components/ReleaseV3";
......
......@@ -268,6 +268,35 @@ STSRow* tGetSubmitBlkNext(SSubmitBlkIter* pIter);
// for debug
int32_t tPrintFixedSchemaSubmitReq(SSubmitReq* pReq, STSchema* pSchema);
struct SSchema {
int8_t type;
int8_t flags;
col_id_t colId;
int32_t bytes;
char name[TSDB_COL_NAME_LEN];
};
typedef struct {
char tbName[TSDB_TABLE_NAME_LEN];
char stbName[TSDB_TABLE_NAME_LEN];
char dbFName[TSDB_DB_FNAME_LEN];
int64_t dbId;
int32_t numOfTags;
int32_t numOfColumns;
int8_t precision;
int8_t tableType;
int32_t sversion;
int32_t tversion;
uint64_t suid;
uint64_t tuid;
int32_t vgId;
int8_t sysInfo;
SSchema* pSchemas;
} STableMetaRsp;
typedef struct {
int32_t code;
int8_t hashMeta;
......@@ -276,6 +305,7 @@ typedef struct {
int32_t numOfRows;
int32_t affectedRows;
int64_t sver;
STableMetaRsp* pMeta;
} SSubmitBlkRsp;
typedef struct {
......@@ -290,6 +320,7 @@ typedef struct {
int32_t tEncodeSSubmitRsp(SEncoder* pEncoder, const SSubmitRsp* pRsp);
int32_t tDecodeSSubmitRsp(SDecoder* pDecoder, SSubmitRsp* pRsp);
void tFreeSSubmitBlkRsp(void* param);
void tFreeSSubmitRsp(SSubmitRsp* pRsp);
#define COL_SMA_ON ((int8_t)0x1)
......@@ -297,13 +328,6 @@ void tFreeSSubmitRsp(SSubmitRsp* pRsp);
#define COL_SET_NULL ((int8_t)0x10)
#define COL_SET_VAL ((int8_t)0x20)
#define COL_IS_SYSINFO ((int8_t)0x40)
struct SSchema {
int8_t type;
int8_t flags;
col_id_t colId;
int32_t bytes;
char name[TSDB_COL_NAME_LEN];
};
#define COL_IS_SET(FLG) (((FLG) & (COL_SET_VAL | COL_SET_NULL)) != 0)
#define COL_CLR_SET(FLG) ((FLG) &= (~(COL_SET_VAL | COL_SET_NULL)))
......@@ -473,6 +497,14 @@ int32_t tSerializeSMCreateStbReq(void* buf, int32_t bufLen, SMCreateStbReq* pReq
int32_t tDeserializeSMCreateStbReq(void* buf, int32_t bufLen, SMCreateStbReq* pReq);
void tFreeSMCreateStbReq(SMCreateStbReq* pReq);
typedef struct {
STableMetaRsp* pMeta;
} SMCreateStbRsp;
int32_t tEncodeSMCreateStbRsp(SEncoder* pEncoder, const SMCreateStbRsp* pRsp);
int32_t tDecodeSMCreateStbRsp(SDecoder* pDecoder, SMCreateStbRsp* pRsp);
void tFreeSMCreateStbRsp(SMCreateStbRsp* pRsp);
typedef struct {
char name[TSDB_TABLE_FNAME_LEN];
int8_t igNotExists;
......@@ -1241,24 +1273,6 @@ typedef struct {
SVgroupInfo vgroups[];
} SVgroupsInfo;
typedef struct {
char tbName[TSDB_TABLE_NAME_LEN];
char stbName[TSDB_TABLE_NAME_LEN];
char dbFName[TSDB_DB_FNAME_LEN];
int64_t dbId;
int32_t numOfTags;
int32_t numOfColumns;
int8_t precision;
int8_t tableType;
int32_t sversion;
int32_t tversion;
uint64_t suid;
uint64_t tuid;
int32_t vgId;
int8_t sysInfo;
SSchema* pSchemas;
} STableMetaRsp;
typedef struct {
STableMetaRsp* pMeta;
} SMAlterStbRsp;
......@@ -1269,7 +1283,7 @@ void tFreeSMAlterStbRsp(SMAlterStbRsp* pRsp);
int32_t tSerializeSTableMetaRsp(void* buf, int32_t bufLen, STableMetaRsp* pRsp);
int32_t tDeserializeSTableMetaRsp(void* buf, int32_t bufLen, STableMetaRsp* pRsp);
void tFreeSTableMetaRsp(STableMetaRsp* pRsp);
void tFreeSTableMetaRsp(void* pRsp);
void tFreeSTableIndexRsp(void* info);
typedef struct {
......@@ -2031,11 +2045,13 @@ int tEncodeSVCreateTbBatchReq(SEncoder* pCoder, const SVCreateTbBatchReq* pReq);
int tDecodeSVCreateTbBatchReq(SDecoder* pCoder, SVCreateTbBatchReq* pReq);
typedef struct {
int32_t code;
int32_t code;
STableMetaRsp* pMeta;
} SVCreateTbRsp, SVUpdateTbRsp;
int tEncodeSVCreateTbRsp(SEncoder* pCoder, const SVCreateTbRsp* pRsp);
int tDecodeSVCreateTbRsp(SDecoder* pCoder, SVCreateTbRsp* pRsp);
void tFreeSVCreateTbRsp(void* param);
int32_t tSerializeSVCreateTbReq(void** buf, SVCreateTbReq* pReq);
void* tDeserializeSVCreateTbReq(void* buf, SVCreateTbReq* pReq);
......
......@@ -215,6 +215,7 @@ void initQueryModuleMsgHandle();
const SSchema* tGetTbnameColumnSchema();
bool tIsValidSchema(struct SSchema* pSchema, int32_t numOfCols, int32_t numOfTags);
int32_t queryCreateCTableMetaFromMsg(STableMetaRsp *msg, SCTableMeta *pMeta);
int32_t queryCreateTableMetaFromMsg(STableMetaRsp* msg, bool isSuperTable, STableMeta** pMeta);
char* jobTaskStatusStr(int32_t status);
......
......@@ -210,6 +210,8 @@ void taosHashSetEqualFp(SHashObj *pHashObj, _equal_fn_t fp);
*/
void taosHashSetFreeFp(SHashObj *pHashObj, _hash_free_fn_t fp);
int64_t taosHashGetCompTimes(SHashObj *pHashObj);
#ifdef __cplusplus
}
#endif
......
......@@ -369,8 +369,9 @@ void launchAsyncQuery(SRequestObj* pRequest, SQuery* pQuery, SMetaData*
int32_t refreshMeta(STscObj* pTscObj, SRequestObj* pRequest);
int32_t updateQnodeList(SAppInstInfo* pInfo, SArray* pNodeList);
void doAsyncQuery(SRequestObj* pRequest, bool forceUpdateMeta);
int32_t removeMeta(STscObj* pTscObj, SArray* tbList); // todo move to clientImpl.c and become a static function
int32_t handleAlterTbExecRes(void* res, struct SCatalog* pCatalog); // todo move to xxx
int32_t removeMeta(STscObj* pTscObj, SArray* tbList);
int32_t handleAlterTbExecRes(void* res, struct SCatalog* pCatalog);
int32_t handleCreateTbExecRes(void* res, SCatalog* pCatalog);
bool qnodeRequired(SRequestObj* pRequest);
#ifdef __cplusplus
......
......@@ -723,6 +723,12 @@ int32_t handleSubmitExecRes(SRequestObj* pRequest, void* res, SCatalog* pCatalog
for (int32_t i = 0; i < pRsp->nBlocks; ++i) {
SSubmitBlkRsp* blk = pRsp->pBlocks + i;
if (blk->pMeta) {
handleCreateTbExecRes(blk->pMeta, pCatalog);
tFreeSTableMetaRsp(blk->pMeta);
taosMemoryFreeClear(blk->pMeta);
}
if (NULL == blk->tblFName || 0 == blk->tblFName[0]) {
continue;
}
......@@ -782,6 +788,10 @@ int32_t handleAlterTbExecRes(void* res, SCatalog* pCatalog) {
return catalogUpdateTableMeta(pCatalog, (STableMetaRsp*)res);
}
int32_t handleCreateTbExecRes(void* res, SCatalog* pCatalog) {
return catalogUpdateTableMeta(pCatalog, (STableMetaRsp*)res);
}
int32_t handleQueryExecRsp(SRequestObj* pRequest) {
if (NULL == pRequest->body.resInfo.execRes.res) {
return TSDB_CODE_SUCCESS;
......@@ -804,6 +814,19 @@ int32_t handleQueryExecRsp(SRequestObj* pRequest) {
code = handleAlterTbExecRes(pRes->res, pCatalog);
break;
}
case TDMT_VND_CREATE_TABLE: {
SArray* pList = (SArray*)pRes->res;
int32_t num = taosArrayGetSize(pList);
for (int32_t i = 0; i < num; ++i) {
void* res = taosArrayGetP(pList, i);
code = handleCreateTbExecRes(res, pCatalog);
}
break;
}
case TDMT_MND_CREATE_STB: {
code = handleCreateTbExecRes(pRes->res, pCatalog);
break;
}
case TDMT_VND_SUBMIT: {
atomic_add_fetch_64((int64_t*)&pAppInfo->summary.insertBytes, pRes->numOfBytes);
......@@ -863,17 +886,13 @@ void schedulerExecCb(SExecResult* pResult, void* param, int32_t code) {
return;
}
if (code == TSDB_CODE_SUCCESS) {
code = handleQueryExecRsp(pRequest);
ASSERT(pRequest->code == TSDB_CODE_SUCCESS);
pRequest->code = code;
}
tscDebug("schedulerExecCb request type %s", TMSG_INFO(pRequest->type));
if (NEED_CLIENT_RM_TBLMETA_REQ(pRequest->type)) {
if (NEED_CLIENT_RM_TBLMETA_REQ(pRequest->type) && NULL == pRequest->body.resInfo.execRes.res) {
removeMeta(pTscObj, pRequest->targetTableList);
}
handleQueryExecRsp(pRequest);
// return to client
pRequest->body.queryFp(pRequest->body.param, pRequest, code);
}
......@@ -934,6 +953,10 @@ SRequestObj* launchQueryImpl(SRequestObj* pRequest, SQuery* pQuery, bool keepQue
qDestroyQuery(pQuery);
}
if (NEED_CLIENT_RM_TBLMETA_REQ(pRequest->type) && NULL == pRequest->body.resInfo.execRes.res) {
removeMeta(pRequest->pTscObj, pRequest->targetTableList);
}
handleQueryExecRsp(pRequest);
if (NULL != pRequest && TSDB_CODE_SUCCESS != code) {
......@@ -1132,10 +1155,6 @@ SRequestObj* execQuery(uint64_t connId, const char* sql, int sqlLen, bool valida
inRetry = true;
} while (retryNum++ < REQUEST_TOTAL_EXEC_TIMES);
if (NEED_CLIENT_RM_TBLMETA_REQ(pRequest->type)) {
removeMeta(pRequest->pTscObj, pRequest->targetTableList);
}
return pRequest;
}
......
......@@ -233,13 +233,36 @@ int32_t processCreateSTableRsp(void* param, SDataBuf* pMsg, int32_t code) {
assert(pMsg != NULL && param != NULL);
SRequestObj* pRequest = param;
taosMemoryFree(pMsg->pData);
if (code != TSDB_CODE_SUCCESS) {
setErrno(pRequest, code);
} else {
SMCreateStbRsp createRsp = {0};
SDecoder coder = {0};
tDecoderInit(&coder, pMsg->pData, pMsg->len);
tDecodeSMCreateStbRsp(&coder, &createRsp);
tDecoderClear(&coder);
pRequest->body.resInfo.execRes.msgType = TDMT_MND_CREATE_STB;
pRequest->body.resInfo.execRes.res = createRsp.pMeta;
}
taosMemoryFree(pMsg->pData);
if (pRequest->body.queryFp != NULL) {
removeMeta(pRequest->pTscObj, pRequest->tableList);
SExecResult* pRes = &pRequest->body.resInfo.execRes;
if (code == TSDB_CODE_SUCCESS) {
SCatalog* pCatalog = NULL;
int32_t ret = catalogGetHandle(pRequest->pTscObj->pAppInfo->clusterId, &pCatalog);
if (pRes->res != NULL) {
ret = handleCreateTbExecRes(pRes->res, pCatalog);
}
if (ret != TSDB_CODE_SUCCESS) {
code = ret;
}
}
pRequest->body.queryFp(pRequest->body.param, pRequest, code);
} else {
tsem_post(&pRequest->body.rspSem);
......
......@@ -420,6 +420,7 @@ static int32_t taosAddServerCfg(SConfig *pCfg) {
if (cfgAddInt32(pCfg, "mqRebalanceInterval", tsMqRebalanceInterval, 1, 10000, 1) != 0) return -1;
if (cfgAddInt32(pCfg, "ttlUnit", tsTtlUnit, 1, 86400 * 365, 1) != 0) return -1;
if (cfgAddInt32(pCfg, "ttlPushInterval", tsTtlPushInterval, 1, 100000, 1) != 0) return -1;
if (cfgAddInt32(pCfg, "uptimeInterval", tsUptimeInterval, 1, 100000, 1) != 0) return -1;
if (cfgAddBool(pCfg, "udf", tsStartUdfd, 0) != 0) return -1;
GRANT_CFG_ADD;
......@@ -567,6 +568,7 @@ static int32_t taosSetServerCfg(SConfig *pCfg) {
tsMqRebalanceInterval = cfgGetItem(pCfg, "mqRebalanceInterval")->i32;
tsTtlUnit = cfgGetItem(pCfg, "ttlUnit")->i32;
tsTtlPushInterval = cfgGetItem(pCfg, "ttlPushInterval")->i32;
tsUptimeInterval = cfgGetItem(pCfg, "uptimeInterval")->i32;
tsStartUdfd = cfgGetItem(pCfg, "udf")->bval;
......
......@@ -3196,12 +3196,16 @@ static int32_t tDecodeSTableMetaRsp(SDecoder *pDecoder, STableMetaRsp *pRsp) {
if (tDecodeI32(pDecoder, &pRsp->vgId) < 0) return -1;
int32_t totalCols = pRsp->numOfTags + pRsp->numOfColumns;
pRsp->pSchemas = taosMemoryMalloc(sizeof(SSchema) * totalCols);
if (pRsp->pSchemas == NULL) return -1;
if (totalCols > 0) {
pRsp->pSchemas = taosMemoryMalloc(sizeof(SSchema) * totalCols);
if (pRsp->pSchemas == NULL) return -1;
for (int32_t i = 0; i < totalCols; ++i) {
SSchema *pSchema = &pRsp->pSchemas[i];
if (tDecodeSSchema(pDecoder, pSchema) < 0) return -1;
for (int32_t i = 0; i < totalCols; ++i) {
SSchema *pSchema = &pRsp->pSchemas[i];
if (tDecodeSSchema(pDecoder, pSchema) < 0) return -1;
}
} else {
pRsp->pSchemas = NULL;
}
return 0;
......@@ -3326,7 +3330,7 @@ int32_t tDeserializeSSTbHbRsp(void *buf, int32_t bufLen, SSTbHbRsp *pRsp) {
return 0;
}
void tFreeSTableMetaRsp(STableMetaRsp *pRsp) { taosMemoryFreeClear(pRsp->pSchemas); }
void tFreeSTableMetaRsp(void *pRsp) { taosMemoryFreeClear(((STableMetaRsp*)pRsp)->pSchemas); }
void tFreeSTableIndexRsp(void *info) {
if (NULL == info) {
......@@ -5092,6 +5096,10 @@ int tEncodeSVCreateTbRsp(SEncoder *pCoder, const SVCreateTbRsp *pRsp) {
if (tStartEncode(pCoder) < 0) return -1;
if (tEncodeI32(pCoder, pRsp->code) < 0) return -1;
if (tEncodeI32(pCoder, pRsp->pMeta ? 1 : 0) < 0) return -1;
if (pRsp->pMeta) {
if (tEncodeSTableMetaRsp(pCoder, pRsp->pMeta) < 0) return -1;
}
tEndEncode(pCoder);
return 0;
......@@ -5102,10 +5110,32 @@ int tDecodeSVCreateTbRsp(SDecoder *pCoder, SVCreateTbRsp *pRsp) {
if (tDecodeI32(pCoder, &pRsp->code) < 0) return -1;
int32_t meta = 0;
if (tDecodeI32(pCoder, &meta) < 0) return -1;
if (meta) {
pRsp->pMeta = taosMemoryCalloc(1, sizeof(STableMetaRsp));
if (NULL == pRsp->pMeta) return -1;
if (tDecodeSTableMetaRsp(pCoder, pRsp->pMeta) < 0) return -1;
} else {
pRsp->pMeta = NULL;
}
tEndDecode(pCoder);
return 0;
}
void tFreeSVCreateTbRsp(void* param) {
if (NULL == param) {
return;
}
SVCreateTbRsp* pRsp = (SVCreateTbRsp*)param;
if (pRsp->pMeta) {
taosMemoryFree(pRsp->pMeta->pSchemas);
taosMemoryFree(pRsp->pMeta);
}
}
// TDMT_VND_DROP_TABLE =================
static int32_t tEncodeSVDropTbReq(SEncoder *pCoder, const SVDropTbReq *pReq) {
if (tStartEncode(pCoder) < 0) return -1;
......@@ -5294,6 +5324,10 @@ static int32_t tEncodeSSubmitBlkRsp(SEncoder *pEncoder, const SSubmitBlkRsp *pBl
if (tEncodeI32v(pEncoder, pBlock->numOfRows) < 0) return -1;
if (tEncodeI32v(pEncoder, pBlock->affectedRows) < 0) return -1;
if (tEncodeI64v(pEncoder, pBlock->sver) < 0) return -1;
if (tEncodeI32(pEncoder, pBlock->pMeta ? 1 : 0) < 0) return -1;
if (pBlock->pMeta) {
if (tEncodeSTableMetaRsp(pEncoder, pBlock->pMeta) < 0) return -1;
}
tEndEncode(pEncoder);
return 0;
......@@ -5311,6 +5345,16 @@ static int32_t tDecodeSSubmitBlkRsp(SDecoder *pDecoder, SSubmitBlkRsp *pBlock) {
if (tDecodeI32v(pDecoder, &pBlock->numOfRows) < 0) return -1;
if (tDecodeI32v(pDecoder, &pBlock->affectedRows) < 0) return -1;
if (tDecodeI64v(pDecoder, &pBlock->sver) < 0) return -1;
int32_t meta = 0;
if (tDecodeI32(pDecoder, &meta) < 0) return -1;
if (meta) {
pBlock->pMeta = taosMemoryCalloc(1, sizeof(STableMetaRsp));
if (NULL == pBlock->pMeta) return -1;
if (tDecodeSTableMetaRsp(pDecoder, pBlock->pMeta) < 0) return -1;
} else {
pBlock->pMeta = NULL;
}
tEndDecode(pDecoder);
return 0;
......@@ -5349,6 +5393,21 @@ int32_t tDecodeSSubmitRsp(SDecoder *pDecoder, SSubmitRsp *pRsp) {
return 0;
}
void tFreeSSubmitBlkRsp(void* param) {
if (NULL == param) {
return;
}
SSubmitBlkRsp* pRsp = (SSubmitBlkRsp*)param;
taosMemoryFree(pRsp->tblFName);
if (pRsp->pMeta) {
taosMemoryFree(pRsp->pMeta->pSchemas);
taosMemoryFree(pRsp->pMeta);
}
}
void tFreeSSubmitRsp(SSubmitRsp *pRsp) {
if (NULL == pRsp) return;
......@@ -5560,6 +5619,60 @@ void tFreeSMAlterStbRsp(SMAlterStbRsp *pRsp) {
}
}
int32_t tEncodeSMCreateStbRsp(SEncoder *pEncoder, const SMCreateStbRsp *pRsp) {
if (tStartEncode(pEncoder) < 0) return -1;
if (tEncodeI32(pEncoder, pRsp->pMeta->pSchemas ? 1 : 0) < 0) return -1;
if (pRsp->pMeta->pSchemas) {
if (tEncodeSTableMetaRsp(pEncoder, pRsp->pMeta) < 0) return -1;
}
tEndEncode(pEncoder);
return 0;
}
int32_t tDecodeSMCreateStbRsp(SDecoder *pDecoder, SMCreateStbRsp *pRsp) {
int32_t meta = 0;
if (tStartDecode(pDecoder) < 0) return -1;
if (tDecodeI32(pDecoder, &meta) < 0) return -1;
if (meta) {
pRsp->pMeta = taosMemoryCalloc(1, sizeof(STableMetaRsp));
if (NULL == pRsp->pMeta) return -1;
if (tDecodeSTableMetaRsp(pDecoder, pRsp->pMeta) < 0) return -1;
}
tEndDecode(pDecoder);
return 0;
}
int32_t tDeserializeSMCreateStbRsp(void *buf, int32_t bufLen, SMCreateStbRsp *pRsp) {
int32_t meta = 0;
SDecoder decoder = {0};
tDecoderInit(&decoder, buf, bufLen);
if (tStartDecode(&decoder) < 0) return -1;
if (tDecodeI32(&decoder, &meta) < 0) return -1;
if (meta) {
pRsp->pMeta = taosMemoryCalloc(1, sizeof(STableMetaRsp));
if (NULL == pRsp->pMeta) return -1;
if (tDecodeSTableMetaRsp(&decoder, pRsp->pMeta) < 0) return -1;
}
tEndDecode(&decoder);
tDecoderClear(&decoder);
return 0;
}
void tFreeSMCreateStbRsp(SMCreateStbRsp *pRsp) {
if (NULL == pRsp) {
return;
}
if (pRsp->pMeta) {
taosMemoryFree(pRsp->pMeta->pSchemas);
taosMemoryFree(pRsp->pMeta);
}
}
int32_t tEncodeSTqOffsetVal(SEncoder *pEncoder, const STqOffsetVal *pOffsetVal) {
if (tEncodeI8(pEncoder, pOffsetVal->type) < 0) return -1;
if (pOffsetVal->type == TMQ_OFFSET__SNAPSHOT_DATA) {
......
......@@ -35,6 +35,7 @@ SDbObj *mndAcquireDbByStb(SMnode *pMnode, const char *stbName);
int32_t mndBuildStbFromReq(SMnode *pMnode, SStbObj *pDst, SMCreateStbReq *pCreate, SDbObj *pDb);
int32_t mndAddStbToTrans(SMnode *pMnode, STrans *pTrans, SDbObj *pDb, SStbObj *pStb);
void mndFreeStb(SStbObj *pStb);
int32_t mndBuildSMCreateStbRsp(SMnode *pMnode, char* dbFName, char* stbFName, void **pCont, int32_t *pLen);
void mndExtractDbNameFromStbFullName(const char *stbFullName, char *dst);
void mndExtractTbNameFromStbFullName(const char *stbFullName, char *dst, int32_t dstSize);
......
......@@ -132,7 +132,7 @@ static void *mndThreadFp(void *param) {
mndCalMqRebalance(pMnode);
}
if (lastTime % (tsTelemInterval * 10) == 1) {
if (lastTime % (tsTelemInterval * 10) == ((tsTelemInterval - 1) * 10)) {
mndPullupTelem(pMnode);
}
......
......@@ -1774,6 +1774,67 @@ static int32_t mndBuildSMAlterStbRsp(SDbObj *pDb, SStbObj *pObj, void **pCont, i
return 0;
}
int32_t mndBuildSMCreateStbRsp(SMnode *pMnode, char* dbFName, char* stbFName, void **pCont, int32_t *pLen) {
int32_t ret = -1;
SDbObj *pDb = mndAcquireDb(pMnode, dbFName);
if (NULL == pDb) {
return -1;
}
SStbObj *pObj = mndAcquireStb(pMnode, stbFName);
if (NULL == pObj) {
goto _OVER;
}
SEncoder ec = {0};
uint32_t contLen = 0;
SMCreateStbRsp stbRsp = {0};
SName name = {0};
tNameFromString(&name, pObj->name, T_NAME_ACCT | T_NAME_DB | T_NAME_TABLE);
stbRsp.pMeta = taosMemoryCalloc(1, sizeof(STableMetaRsp));
if (NULL == stbRsp.pMeta) {
terrno = TSDB_CODE_OUT_OF_MEMORY;
goto _OVER;
}
ret = mndBuildStbSchemaImp(pDb, pObj, name.tname, stbRsp.pMeta);
if (ret) {
tFreeSMCreateStbRsp(&stbRsp);
goto _OVER;
}
tEncodeSize(tEncodeSMCreateStbRsp, &stbRsp, contLen, ret);
if (ret) {
tFreeSMCreateStbRsp(&stbRsp);
goto _OVER;
}
void *cont = taosMemoryMalloc(contLen);
tEncoderInit(&ec, cont, contLen);
tEncodeSMCreateStbRsp(&ec, &stbRsp);
tEncoderClear(&ec);
tFreeSMCreateStbRsp(&stbRsp);
*pCont = cont;
*pLen = contLen;
ret = 0;
_OVER:
if (pObj) {
mndReleaseStb(pMnode, pObj);
}
if (pDb) {
mndReleaseDb(pMnode, pDb);
}
return ret;
}
static int32_t mndAlterStbImp(SMnode *pMnode, SRpcMsg *pReq, SDbObj *pDb, SStbObj *pStb, bool needRsp,
void *alterOriData, int32_t alterOriDataLen) {
int32_t code = -1;
......
......@@ -17,6 +17,7 @@
#include "mndTrans.h"
#include "mndConsumer.h"
#include "mndDb.h"
#include "mndStb.h"
#include "mndPrivilege.h"
#include "mndShow.h"
#include "mndSync.h"
......@@ -900,15 +901,6 @@ static void mndTransSendRpcRsp(SMnode *pMnode, STrans *pTrans) {
}
SRpcMsg rspMsg = {.code = code, .info = *pInfo};
if (pTrans->rpcRspLen != 0) {
void *rpcCont = rpcMallocCont(pTrans->rpcRspLen);
if (rpcCont != NULL) {
memcpy(rpcCont, pTrans->rpcRsp, pTrans->rpcRspLen);
rspMsg.pCont = rpcCont;
rspMsg.contLen = pTrans->rpcRspLen;
}
}
if (pTrans->originRpcType == TDMT_MND_CREATE_DB) {
mDebug("trans:%d, origin msgtype:%s", pTrans->id, TMSG_INFO(pTrans->originRpcType));
SDbObj *pDb = mndAcquireDb(pMnode, pTrans->dbname1);
......@@ -924,6 +916,21 @@ static void mndTransSendRpcRsp(SMnode *pMnode, STrans *pTrans) {
}
}
mndReleaseDb(pMnode, pDb);
} else if (pTrans->originRpcType == TDMT_MND_CREATE_STB) {
void *pCont = NULL;
int32_t contLen = 0;
if (0 == mndBuildSMCreateStbRsp(pMnode, pTrans->dbname1, pTrans->dbname2, &pCont, &contLen) != 0) {
mndTransSetRpcRsp(pTrans, pCont, contLen);
}
}
if (pTrans->rpcRspLen != 0) {
void *rpcCont = rpcMallocCont(pTrans->rpcRspLen);
if (rpcCont != NULL) {
memcpy(rpcCont, pTrans->rpcRsp, pTrans->rpcRspLen);
rspMsg.pCont = rpcCont;
rspMsg.contLen = pTrans->rpcRspLen;
}
}
tmsgSendRsp(&rspMsg);
......
......@@ -63,7 +63,7 @@ void vnodeGetInfo(SVnode *pVnode, const char **dbname, int32_t *vgId);
int32_t vnodeProcessCreateTSma(SVnode *pVnode, void *pCont, uint32_t contLen);
int32_t vnodeGetAllTableList(SVnode *pVnode, uint64_t uid, SArray *list);
int32_t vnodeGetCtbIdList(SVnode *pVnode, int64_t suid, SArray *list);
int32_t vnodeGetStbIdList(SVnode *pVnode, int64_t suid, SArray* list);
int32_t vnodeGetStbIdList(SVnode *pVnode, int64_t suid, SArray *list);
void *vnodeGetIdx(SVnode *pVnode);
void *vnodeGetIvtIdx(SVnode *pVnode);
......@@ -96,7 +96,7 @@ int32_t metaGetTableTags(SMeta *pMeta, uint64_t suid, SArray *uidList, SHash
int32_t metaReadNext(SMetaReader *pReader);
const void *metaGetTableTagVal(void *tag, int16_t type, STagVal *tagVal);
int metaGetTableNameByUid(void *meta, uint64_t uid, char *tbName);
bool metaIsTableExist(SMeta *pMeta, tb_uid_t uid);
bool metaIsTableExist(SMeta *pMeta, tb_uid_t uid);
typedef struct SMetaFltParam {
tb_uid_t suid;
......@@ -224,6 +224,7 @@ typedef struct {
int64_t numOfSTables;
int64_t numOfCTables;
int64_t numOfNTables;
int64_t numOfNTimeSeries;
int64_t numOfTimeSeries;
int64_t pointsWritten;
int64_t totalStorage;
......
......@@ -102,7 +102,7 @@ int metaCommit(SMeta* pMeta);
int metaCreateSTable(SMeta* pMeta, int64_t version, SVCreateStbReq* pReq);
int metaAlterSTable(SMeta* pMeta, int64_t version, SVCreateStbReq* pReq);
int metaDropSTable(SMeta* pMeta, int64_t verison, SVDropStbReq* pReq, SArray* tbUidList);
int metaCreateTable(SMeta* pMeta, int64_t version, SVCreateTbReq* pReq);
int metaCreateTable(SMeta* pMeta, int64_t version, SVCreateTbReq* pReq, STableMetaRsp **pMetaRsp);
int metaDropTable(SMeta* pMeta, int64_t version, SVDropTbReq* pReq, SArray* tbUids);
int metaTtlDropTable(SMeta* pMeta, int64_t ttl, SArray* tbUids);
int metaAlterTable(SMeta* pMeta, int64_t version, SVAlterTbReq* pReq, STableMetaRsp* pMetaRsp);
......
......@@ -619,7 +619,7 @@ int64_t metaGetTimeSeriesNum(SMeta *pMeta) {
vnodeGetTimeSeriesNum(pMeta->pVnode, &num);
pMeta->pVnode->config.vndStats.numOfTimeSeries = num;
return pMeta->pVnode->config.vndStats.numOfTimeSeries;
return pMeta->pVnode->config.vndStats.numOfTimeSeries + pMeta->pVnode->config.vndStats.numOfNTimeSeries;
}
typedef struct {
......
......@@ -367,7 +367,7 @@ int metaAlterSTable(SMeta *pMeta, int64_t version, SVCreateStbReq *pReq) {
return 0;
}
int metaCreateTable(SMeta *pMeta, int64_t version, SVCreateTbReq *pReq) {
int metaCreateTable(SMeta *pMeta, int64_t version, SVCreateTbReq *pReq, STableMetaRsp **pMetaRsp) {
SMetaEntry me = {0};
SMetaReader mr = {0};
......@@ -423,10 +423,26 @@ int metaCreateTable(SMeta *pMeta, int64_t version, SVCreateTbReq *pReq) {
me.ntbEntry.ncid = me.ntbEntry.schemaRow.pSchema[me.ntbEntry.schemaRow.nCols - 1].colId + 1;
++pMeta->pVnode->config.vndStats.numOfNTables;
pMeta->pVnode->config.vndStats.numOfNTimeSeries += me.ntbEntry.schemaRow.nCols - 1;
}
if (metaHandleEntry(pMeta, &me) < 0) goto _err;
if (pMetaRsp) {
*pMetaRsp = taosMemoryCalloc(1, sizeof(STableMetaRsp));
if (*pMetaRsp) {
if (me.type == TSDB_CHILD_TABLE) {
(*pMetaRsp)->tableType = TSDB_CHILD_TABLE;
(*pMetaRsp)->tuid = pReq->uid;
(*pMetaRsp)->suid = pReq->ctb.suid;
strcpy((*pMetaRsp)->tbName, pReq->name);
} else {
metaUpdateMetaRsp(pReq->uid, pReq->name, &pReq->ntb.schemaRow, *pMetaRsp);
}
}
}
metaDebug("vgId:%d, table:%s uid %" PRId64 " is created, type:%" PRId8, TD_VID(pMeta->pVnode), pReq->name, pReq->uid,
pReq->type);
return 0;
......@@ -562,6 +578,7 @@ static int metaDropTableByUid(SMeta *pMeta, tb_uid_t uid, int *type) {
// drop schema.db (todo)
--pMeta->pVnode->config.vndStats.numOfNTables;
pMeta->pVnode->config.vndStats.numOfNTimeSeries -= e.ntbEntry.schemaRow.nCols - 1;
} else if (e.type == TSDB_SUPER_TABLE) {
tdbTbDelete(pMeta->pSuidIdx, &e.uid, sizeof(tb_uid_t), &pMeta->txn);
// drop schema.db (todo)
......@@ -664,6 +681,8 @@ static int metaAlterTableColumn(SMeta *pMeta, int64_t version, SVAlterTbReq *pAl
pSchema->pSchema[entry.ntbEntry.schemaRow.nCols - 1].flags = pAlterTbReq->flags;
pSchema->pSchema[entry.ntbEntry.schemaRow.nCols - 1].colId = entry.ntbEntry.ncid++;
strcpy(pSchema->pSchema[entry.ntbEntry.schemaRow.nCols - 1].name, pAlterTbReq->colName);
++pMeta->pVnode->config.vndStats.numOfNTimeSeries;
break;
case TSDB_ALTER_TABLE_DROP_COLUMN:
if (pColumn == NULL) {
......@@ -684,6 +703,8 @@ static int metaAlterTableColumn(SMeta *pMeta, int64_t version, SVAlterTbReq *pAl
memmove(pColumn, pColumn + 1, tlen);
}
pSchema->nCols--;
--pMeta->pVnode->config.vndStats.numOfNTimeSeries;
break;
case TSDB_ALTER_TABLE_UPDATE_COLUMN_BYTES:
if (pColumn == NULL) {
......
......@@ -117,6 +117,7 @@ int vnodeEncodeConfig(const void *pObj, SJson *pJson) {
if (tjsonAddIntegerToObject(pJson, "vndStats.ctables", pCfg->vndStats.numOfCTables) < 0) return -1;
if (tjsonAddIntegerToObject(pJson, "vndStats.ntables", pCfg->vndStats.numOfNTables) < 0) return -1;
if (tjsonAddIntegerToObject(pJson, "vndStats.timeseries", pCfg->vndStats.numOfTimeSeries) < 0) return -1;
if (tjsonAddIntegerToObject(pJson, "vndStats.ntimeseries", pCfg->vndStats.numOfNTimeSeries) < 0) return -1;
SJson *pNodeInfoArr = tjsonCreateArray();
tjsonAddItemToObject(pJson, "syncCfg.nodeInfo", pNodeInfoArr);
......@@ -224,6 +225,8 @@ int vnodeDecodeConfig(const SJson *pJson, void *pObj) {
if (code < 0) return -1;
tjsonGetNumberValue(pJson, "vndStats.timeseries", pCfg->vndStats.numOfTimeSeries, code);
if (code < 0) return -1;
tjsonGetNumberValue(pJson, "vndStats.ntimeseries", pCfg->vndStats.numOfNTimeSeries, code);
if (code < 0) return -1;
SJson *pNodeInfoArr = tjsonGetObjectItem(pJson, "syncCfg.nodeInfo");
int arraySize = tjsonGetArraySize(pNodeInfoArr);
......
......@@ -368,6 +368,10 @@ void smaHandleRes(void *pVnode, int64_t smaId, const SArray *data) {
}
void vnodeUpdateMetaRsp(SVnode *pVnode, STableMetaRsp *pMetaRsp) {
if (NULL == pMetaRsp) {
return;
}
strcpy(pMetaRsp->dbFName, pVnode->config.dbname);
pMetaRsp->dbId = pVnode->config.dbId;
pMetaRsp->vgId = TD_VID(pVnode);
......@@ -512,7 +516,7 @@ static int32_t vnodeProcessCreateTbReq(SVnode *pVnode, int64_t version, void *pR
}
// do create table
if (metaCreateTable(pVnode->pMeta, version, pCreateReq) < 0) {
if (metaCreateTable(pVnode->pMeta, version, pCreateReq, &cRsp.pMeta) < 0) {
if (pCreateReq->flags & TD_CREATE_IF_NOT_EXISTS && terrno == TSDB_CODE_TDB_TABLE_ALREADY_EXIST) {
cRsp.code = TSDB_CODE_SUCCESS;
} else {
......@@ -522,6 +526,7 @@ static int32_t vnodeProcessCreateTbReq(SVnode *pVnode, int64_t version, void *pR
cRsp.code = TSDB_CODE_SUCCESS;
tdFetchTbUidList(pVnode->pSma, &pStore, pCreateReq->ctb.suid, pCreateReq->uid);
taosArrayPush(tbUids, &pCreateReq->uid);
vnodeUpdateMetaRsp(pVnode, cRsp.pMeta);
}
taosArrayPush(rsp.pArray, &cRsp);
......@@ -550,7 +555,7 @@ _exit:
pCreateReq = req.pReqs + iReq;
taosArrayDestroy(pCreateReq->ctb.tagName);
}
taosArrayDestroy(rsp.pArray);
taosArrayDestroyEx(rsp.pArray, tFreeSVCreateTbRsp);
taosArrayDestroy(tbUids);
tDecoderClear(&decoder);
tEncoderClear(&encoder);
......@@ -862,7 +867,7 @@ static int32_t vnodeProcessSubmitReq(SVnode *pVnode, int64_t version, void *pReq
goto _exit;
}
if (metaCreateTable(pVnode->pMeta, version, &createTbReq) < 0) {
if (metaCreateTable(pVnode->pMeta, version, &createTbReq, &submitBlkRsp.pMeta) < 0) {
if (terrno != TSDB_CODE_TDB_TABLE_ALREADY_EXIST) {
submitBlkRsp.code = terrno;
pRsp->code = terrno;
......@@ -870,6 +875,10 @@ static int32_t vnodeProcessSubmitReq(SVnode *pVnode, int64_t version, void *pReq
taosArrayDestroy(createTbReq.ctb.tagName);
goto _exit;
}
} else {
if (NULL != submitBlkRsp.pMeta) {
vnodeUpdateMetaRsp(pVnode, submitBlkRsp.pMeta);
}
}
taosArrayPush(newTbUids, &createTbReq.uid);
......@@ -913,11 +922,7 @@ _exit:
tEncodeSSubmitRsp(&encoder, &submitRsp);
tEncoderClear(&encoder);
for (int32_t i = 0; i < taosArrayGetSize(submitRsp.pArray); i++) {
taosMemoryFree(((SSubmitBlkRsp *)taosArrayGet(submitRsp.pArray, i))[0].tblFName);
}
taosArrayDestroy(submitRsp.pArray);
taosArrayDestroyEx(submitRsp.pArray, tFreeSSubmitBlkRsp);
// TODO: the partial success scenario and the error case
// => If partial success, extract the success submitted rows and reconstruct a new submit msg, and push to level
......
......@@ -270,13 +270,22 @@ int32_t ctgUpdateTbMeta(SCatalog* pCtg, STableMetaRsp* rspMsg, bool syncOp) {
int32_t code = 0;
strcpy(output->dbFName, rspMsg->dbFName);
strcpy(output->tbName, rspMsg->tbName);
output->dbId = rspMsg->dbId;
SET_META_TYPE_TABLE(output->metaType);
if (TSDB_CHILD_TABLE == rspMsg->tableType && NULL == rspMsg->pSchemas) {
strcpy(output->ctbName, rspMsg->tbName);
CTG_ERR_JRET(queryCreateTableMetaFromMsg(rspMsg, rspMsg->tableType == TSDB_SUPER_TABLE, &output->tbMeta));
SET_META_TYPE_CTABLE(output->metaType);
CTG_ERR_JRET(queryCreateCTableMetaFromMsg(rspMsg, &output->ctbMeta));
} else {
strcpy(output->tbName, rspMsg->tbName);
SET_META_TYPE_TABLE(output->metaType);
CTG_ERR_JRET(queryCreateTableMetaFromMsg(rspMsg, rspMsg->tableType == TSDB_SUPER_TABLE, &output->tbMeta));
}
CTG_ERR_JRET(ctgUpdateTbMetaEnqueue(pCtg, output, syncOp));
......
......@@ -135,7 +135,12 @@ int32_t nodesNodeToSQL(SNode *pNode, char *buf, int32_t bufSize, int32_t *len) {
NODES_ERR_RET(TSDB_CODE_QRY_APP_ERROR);
}
*len += snprintf(buf + *len, bufSize - *len, "%s", t);
int32_t tlen = strlen(t);
if (tlen > 32) {
*len += snprintf(buf + *len, bufSize - *len, "%.*s...%s", 32, t, t + tlen - 1);
} else {
*len += snprintf(buf + *len, bufSize - *len, "%s", t);
}
taosMemoryFree(t);
return TSDB_CODE_SUCCESS;
......@@ -199,12 +204,17 @@ int32_t nodesNodeToSQL(SNode *pNode, char *buf, int32_t bufSize, int32_t *len) {
SNodeListNode *pListNode = (SNodeListNode *)pNode;
SNode *node = NULL;
bool first = true;
int32_t num = 0;
*len += snprintf(buf + *len, bufSize - *len, "(");
FOREACH(node, pListNode->pNodeList) {
if (!first) {
*len += snprintf(buf + *len, bufSize - *len, ", ");
if (++num >= 10) {
*len += snprintf(buf + *len, bufSize - *len, "...");
break;
}
}
NODES_ERR_RET(nodesNodeToSQL(node, buf, bufSize, len));
first = false;
......
......@@ -213,15 +213,25 @@ SSchema createSchema(int8_t type, int32_t bytes, col_id_t colId, const char* nam
return s;
}
void freeSTableMetaRspPointer(void *p) {
tFreeSTableMetaRsp(*(void**)p);
taosMemoryFreeClear(*(void**)p);
}
void destroyQueryExecRes(SExecResult* pRes) {
if (NULL == pRes || NULL == pRes->res) {
return;
}
switch (pRes->msgType) {
case TDMT_VND_CREATE_TABLE: {
taosArrayDestroyEx((SArray*)pRes->res, freeSTableMetaRspPointer);
break;
}
case TDMT_MND_CREATE_STB:
case TDMT_VND_ALTER_TABLE:
case TDMT_MND_ALTER_STB: {
tFreeSTableMetaRsp((STableMetaRsp*)pRes->res);
tFreeSTableMetaRsp(pRes->res);
taosMemoryFreeClear(pRes->res);
break;
}
......
......@@ -354,6 +354,19 @@ static int32_t queryConvertTableMetaMsg(STableMetaRsp *pMetaMsg) {
return TSDB_CODE_SUCCESS;
}
int32_t queryCreateCTableMetaFromMsg(STableMetaRsp *msg, SCTableMeta *pMeta) {
pMeta->vgId = msg->vgId;
pMeta->tableType = msg->tableType;
pMeta->uid = msg->tuid;
pMeta->suid = msg->suid;
qDebug("ctable %s uid %" PRIx64 " meta returned, type %d vgId:%d db %s suid %" PRIx64 ,
msg->tbName, pMeta->uid, pMeta->tableType, pMeta->vgId, msg->dbFName, pMeta->suid);
return TSDB_CODE_SUCCESS;
}
int32_t queryCreateTableMetaFromMsg(STableMetaRsp *msg, bool isStb, STableMeta **pMeta) {
int32_t total = msg->numOfColumns + msg->numOfTags;
int32_t metaSize = sizeof(STableMeta) + sizeof(SSchema) * total;
......
......@@ -102,15 +102,30 @@ int32_t schHandleResponseMsg(SSchJob *pJob, SSchTask *pTask, int32_t execId, SDa
tDecoderInit(&coder, msg, msgSize);
code = tDecodeSVCreateTbBatchRsp(&coder, &batchRsp);
if (TSDB_CODE_SUCCESS == code && batchRsp.nRsps > 0) {
SCH_LOCK(SCH_WRITE, &pJob->resLock);
if (NULL == pJob->execRes.res) {
pJob->execRes.res = taosArrayInit(batchRsp.nRsps, POINTER_BYTES);
pJob->execRes.msgType = TDMT_VND_CREATE_TABLE;
}
for (int32_t i = 0; i < batchRsp.nRsps; ++i) {
SVCreateTbRsp *rsp = batchRsp.pRsps + i;
if (rsp->pMeta) {
taosArrayPush((SArray*)pJob->execRes.res, &rsp->pMeta);
}
if (TSDB_CODE_SUCCESS != rsp->code) {
code = rsp->code;
tDecoderClear(&coder);
SCH_ERR_JRET(code);
}
}
SCH_UNLOCK(SCH_WRITE, &pJob->resLock);
if (taosArrayGetSize((SArray*)pJob->execRes.res) <= 0) {
taosArrayDestroy((SArray*)pJob->execRes.res);
pJob->execRes.res = NULL;
}
}
tDecoderClear(&coder);
SCH_ERR_JRET(code);
}
......
......@@ -934,6 +934,8 @@ static int tdbFetchOvflPage(SPgno *pPgno, SPage **ppOfp, TXN *pTxn, SBTree *pBt)
return -1;
}
tdbPCacheRelease(pBt->pPager->pCache, *ppOfp, pTxn);
return ret;
}
......@@ -1277,6 +1279,8 @@ static int tdbBtreeDecodePayload(SPage *pPage, const SCell *pCell, int nHeader,
nLeft -= bytes;
memcpy(&pgno, ofpCell + bytes, sizeof(pgno));
tdbPCacheRelease(pBt->pPager->pCache, ofp, pTxn);
}
} else {
int nLeftKey = kLen;
......@@ -1336,6 +1340,8 @@ static int tdbBtreeDecodePayload(SPage *pPage, const SCell *pCell, int nHeader,
memcpy(&pgno, ofpCell + bytes, sizeof(pgno));
tdbPCacheRelease(pBt->pPager->pCache, ofp, pTxn);
nLeftKey -= bytes;
nLeft -= bytes;
}
......@@ -1374,6 +1380,8 @@ static int tdbBtreeDecodePayload(SPage *pPage, const SCell *pCell, int nHeader,
memcpy(&pgno, ofpCell + vLen - nLeft + bytes, sizeof(pgno));
tdbPCacheRelease(pBt->pPager->pCache, ofp, pTxn);
nLeft -= bytes;
}
}
......
......@@ -111,6 +111,7 @@ void tdbPCacheRelease(SPCache *pCache, SPage *pPage, TXN *pTxn) {
tdbPCacheLock(pCache);
nRef = tdbUnrefPage(pPage);
tdbDebug("pcache/release page %p/%d/%d/%d", pPage, TDB_PAGE_PGNO(pPage), pPage->id, nRef);
if (nRef == 0) {
// test the nRef again to make sure
// it is safe th handle the page
......@@ -212,7 +213,8 @@ static SPage *tdbPCacheFetchImpl(SPCache *pCache, const SPgid *pPgid, TXN *pTxn)
pPage->pPager = pPageH->pPager;
memcpy(pPage->pData, pPageH->pData, pPage->pageSize);
tdbDebug("pcache/pPageH: %p %d %p %p", pPageH, pPageH->pPageHdr - pPageH->pData, pPageH->xCellSize, pPage);
tdbDebug("pcache/pPageH: %p %d %p %p %d", pPageH, pPageH->pPageHdr - pPageH->pData, pPageH->xCellSize, pPage,
TDB_PAGE_PGNO(pPageH));
tdbPageInit(pPage, pPageH->pPageHdr - pPageH->pData, pPageH->xCellSize);
pPage->kLen = pPageH->kLen;
pPage->vLen = pPageH->vLen;
......@@ -243,7 +245,7 @@ static void tdbPCachePinPage(SPCache *pCache, SPage *pPage) {
pCache->nRecyclable--;
// printf("pin page %d pgno %d pPage %p\n", pPage->id, TDB_PAGE_PGNO(pPage), pPage);
tdbDebug("pin page %d", pPage->id);
tdbDebug("pcache/pin page %p/%d/%d", pPage, TDB_PAGE_PGNO(pPage), pPage->id);
}
}
......@@ -264,29 +266,23 @@ static void tdbPCacheUnpinPage(SPCache *pCache, SPage *pPage) {
pCache->nRecyclable++;
// printf("unpin page %d pgno %d pPage %p\n", pPage->id, TDB_PAGE_PGNO(pPage), pPage);
tdbDebug("unpin page %d", pPage->id);
tdbDebug("pcache/unpin page %p/%d/%d", pPage, TDB_PAGE_PGNO(pPage), pPage->id);
}
static void tdbPCacheRemovePageFromHash(SPCache *pCache, SPage *pPage) {
uint32_t h = tdbPCachePageHash(&(pPage->pgid)) % pCache->nHash;
SPage **ppPage = &(pCache->pgHash[h]);
if (*ppPage == pPage) {
pCache->pgHash[h] = pPage->pHashNext;
} else {
for (; (*ppPage) && (*ppPage)->pHashNext != pPage; ppPage = &((*ppPage)->pHashNext))
;
if (*ppPage) {
(*ppPage)->pHashNext = pPage->pHashNext;
}
}
for (; (*ppPage) && *ppPage != pPage; ppPage = &((*ppPage)->pHashNext))
;
if (*ppPage) {
pPage->pHashNext = NULL;
--pCache->nPage;
*ppPage = pPage->pHashNext;
pCache->nPage--;
// printf("rmv page %d to hash, pgno %d, pPage %p\n", pPage->id, TDB_PAGE_PGNO(pPage), pPage);
}
tdbDebug("remove page %p/%d from hash", pPage, pPage->id);
tdbDebug("pcache/remove page %p/%d/%d from hash %" PRIu32, pPage, TDB_PAGE_PGNO(pPage), pPage->id, h);
}
static void tdbPCacheAddPageToHash(SPCache *pCache, SPage *pPage) {
......@@ -298,7 +294,7 @@ static void tdbPCacheAddPageToHash(SPCache *pCache, SPage *pPage) {
pCache->nPage++;
// printf("add page %d to hash, pgno %d, pPage %p\n", pPage->id, TDB_PAGE_PGNO(pPage), pPage);
tdbDebug("add page %p/%d to hash", pPage, pPage->id);
tdbDebug("pcache/add page %p/%d/%d to hash %" PRIu32, pPage, TDB_PAGE_PGNO(pPage), pPage->id, h);
}
static int tdbPCacheOpenImpl(SPCache *pCache) {
......
......@@ -68,6 +68,8 @@ int tdbPageCreate(int pageSize, SPage **ppPage, void *(*xMalloc)(void *, size_t)
}
*ppPage = pPage;
tdbDebug("page/create: %p %p", pPage, xMalloc);
return 0;
}
......
......@@ -21,7 +21,7 @@
// the add ref count operation may trigger the warning if the reference count is greater than the MAX_WARNING_REF_COUNT
#define MAX_WARNING_REF_COUNT 10000
#define HASH_MAX_CAPACITY (1024 * 1024 * 16)
#define HASH_MAX_CAPACITY (1024 * 1024 * 1024)
#define HASH_DEFAULT_LOAD_FACTOR (0.75)
#define HASH_INDEX(v, c) ((v) & ((c)-1))
......@@ -67,6 +67,7 @@ struct SHashObj {
bool enableUpdate; // enable update
SArray *pMemBlock; // memory block allocated for SHashEntry
_hash_before_fn_t callbackFp; // function invoked before return the value to caller
int64_t compTimes;
};
/*
......@@ -146,6 +147,7 @@ static FORCE_INLINE SHashNode *doSearchInEntryList(SHashObj *pHashObj, SHashEntr
uint32_t hashVal) {
SHashNode *pNode = pe->next;
while (pNode) {
atomic_add_fetch_64(&pHashObj->compTimes, 1);
if ((pNode->keyLen == keyLen) && ((*(pHashObj->equalFp))(GET_HASH_NODE_KEY(pNode), key, keyLen) == 0) &&
pNode->removed == 0) {
assert(pNode->hashVal == hashVal);
......@@ -886,3 +888,7 @@ void *taosHashAcquire(SHashObj *pHashObj, const void *key, size_t keyLen) {
}
void taosHashRelease(SHashObj *pHashObj, void *p) { taosHashCancelIterate(pHashObj, p); }
int64_t taosHashGetCompTimes(SHashObj *pHashObj) { return atomic_load_64(&pHashObj->compTimes); }
......@@ -197,6 +197,201 @@ void acquireRleaseTest() {
taosMemoryFreeClear(data.p);
}
void perfTest() {
SHashObj* hash1h = (SHashObj*) taosHashInit(100, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BINARY), true, HASH_NO_LOCK);
SHashObj* hash1s = (SHashObj*) taosHashInit(1000, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BINARY), true, HASH_NO_LOCK);
SHashObj* hash10s = (SHashObj*) taosHashInit(10000, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BINARY), true, HASH_NO_LOCK);
SHashObj* hash100s = (SHashObj*) taosHashInit(100000, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BINARY), true, HASH_NO_LOCK);
SHashObj* hash1m = (SHashObj*) taosHashInit(1000000, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BINARY), true, HASH_NO_LOCK);
SHashObj* hash10m = (SHashObj*) taosHashInit(10000000, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BINARY), true, HASH_NO_LOCK);
SHashObj* hash100m = (SHashObj*) taosHashInit(100000000, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BINARY), true, HASH_NO_LOCK);
char *name = (char*)taosMemoryCalloc(50000000, 9);
for (int64_t i = 0; i < 50000000; ++i) {
sprintf(name + i * 9, "t%08d", i);
}
for (int64_t i = 0; i < 50; ++i) {
taosHashPut(hash1h, name + i * 9, 9, &i, sizeof(i));
}
for (int64_t i = 0; i < 500; ++i) {
taosHashPut(hash1s, name + i * 9, 9, &i, sizeof(i));
}
for (int64_t i = 0; i < 5000; ++i) {
taosHashPut(hash10s, name + i * 9, 9, &i, sizeof(i));
}
for (int64_t i = 0; i < 50000; ++i) {
taosHashPut(hash100s, name + i * 9, 9, &i, sizeof(i));
}
for (int64_t i = 0; i < 500000; ++i) {
taosHashPut(hash1m, name + i * 9, 9, &i, sizeof(i));
}
for (int64_t i = 0; i < 5000000; ++i) {
taosHashPut(hash10m, name + i * 9, 9, &i, sizeof(i));
}
for (int64_t i = 0; i < 50000000; ++i) {
taosHashPut(hash100m, name + i * 9, 9, &i, sizeof(i));
}
int64_t start1h = taosGetTimestampMs();
int64_t start1hCt = taosHashGetCompTimes(hash1h);
for (int64_t i = 0; i < 10000000; ++i) {
ASSERT(taosHashGet(hash1h, name + (i % 50) * 9, 9));
}
int64_t end1h = taosGetTimestampMs();
int64_t end1hCt = taosHashGetCompTimes(hash1h);
int64_t start1s = taosGetTimestampMs();
int64_t start1sCt = taosHashGetCompTimes(hash1s);
for (int64_t i = 0; i < 10000000; ++i) {
ASSERT(taosHashGet(hash1s, name + (i % 500) * 9, 9));
}
int64_t end1s = taosGetTimestampMs();
int64_t end1sCt = taosHashGetCompTimes(hash1s);
int64_t start10s = taosGetTimestampMs();
int64_t start10sCt = taosHashGetCompTimes(hash10s);
for (int64_t i = 0; i < 10000000; ++i) {
ASSERT(taosHashGet(hash10s, name + (i % 5000) * 9, 9));
}
int64_t end10s = taosGetTimestampMs();
int64_t end10sCt = taosHashGetCompTimes(hash10s);
int64_t start100s = taosGetTimestampMs();
int64_t start100sCt = taosHashGetCompTimes(hash100s);
for (int64_t i = 0; i < 10000000; ++i) {
ASSERT(taosHashGet(hash100s, name + (i % 50000) * 9, 9));
}
int64_t end100s = taosGetTimestampMs();
int64_t end100sCt = taosHashGetCompTimes(hash100s);
int64_t start1m = taosGetTimestampMs();
int64_t start1mCt = taosHashGetCompTimes(hash1m);
for (int64_t i = 0; i < 10000000; ++i) {
ASSERT(taosHashGet(hash1m, name + (i % 500000) * 9, 9));
}
int64_t end1m = taosGetTimestampMs();
int64_t end1mCt = taosHashGetCompTimes(hash1m);
int64_t start10m = taosGetTimestampMs();
int64_t start10mCt = taosHashGetCompTimes(hash10m);
for (int64_t i = 0; i < 10000000; ++i) {
ASSERT(taosHashGet(hash10m, name + (i % 5000000) * 9, 9));
}
int64_t end10m = taosGetTimestampMs();
int64_t end10mCt = taosHashGetCompTimes(hash10m);
int64_t start100m = taosGetTimestampMs();
int64_t start100mCt = taosHashGetCompTimes(hash100m);
for (int64_t i = 0; i < 10000000; ++i) {
ASSERT(taosHashGet(hash100m, name + (i % 50000000) * 9, 9));
}
int64_t end100m = taosGetTimestampMs();
int64_t end100mCt = taosHashGetCompTimes(hash100m);
SArray *sArray[1000] = {0};
for (int64_t i = 0; i < 1000; ++i) {
sArray[i] = taosArrayInit(100000, 9);
}
int64_t cap = 4;
while (cap < 100000000) cap = (cap << 1u);
_hash_fn_t hashFp = taosGetDefaultHashFunction(TSDB_DATA_TYPE_BINARY);
int32_t slotR = cap / 1000 + 1;
for (int64_t i = 0; i < 10000000; ++i) {
char* p = name + (i % 50000000) * 9;
uint32_t v = (*hashFp)(p, 9);
taosArrayPush(sArray[(v%cap)/slotR], p);
}
SArray *slArray = taosArrayInit(100000000, 9);
for (int64_t i = 0; i < 1000; ++i) {
int32_t num = taosArrayGetSize(sArray[i]);
SArray* pArray = sArray[i];
for (int64_t m = 0; m < num; ++m) {
char* p = (char*)taosArrayGet(pArray, m);
ASSERT(taosArrayPush(slArray, p));
}
}
int64_t start100mS = taosGetTimestampMs();
int64_t start100mSCt = taosHashGetCompTimes(hash100m);
int32_t num = taosArrayGetSize(slArray);
for (int64_t i = 0; i < num; ++i) {
ASSERT(taosHashGet(hash100m, (char*)TARRAY_GET_ELEM(slArray, i), 9));
}
int64_t end100mS = taosGetTimestampMs();
int64_t end100mSCt = taosHashGetCompTimes(hash100m);
for (int64_t i = 0; i < 1000; ++i) {
taosArrayDestroy(sArray[i]);
}
taosArrayDestroy(slArray);
printf("1h \t %" PRId64 "ms,%" PRId64 "\n", end1h - start1h, end1hCt - start1hCt);
printf("1s \t %" PRId64 "ms,%" PRId64 "\n", end1s - start1s, end1sCt - start1sCt);
printf("10s \t %" PRId64 "ms,%" PRId64 "\n", end10s - start10s, end10sCt - start10sCt);
printf("100s \t %" PRId64 "ms,%" PRId64 "\n", end100s - start100s, end100sCt - start100sCt);
printf("1m \t %" PRId64 "ms,%" PRId64 "\n", end1m - start1m, end1mCt - start1mCt);
printf("10m \t %" PRId64 "ms,%" PRId64 "\n", end10m - start10m, end10mCt - start10mCt);
printf("100m \t %" PRId64 "ms,%" PRId64 "\n", end100m - start100m, end100mCt - start100mCt);
printf("100mS \t %" PRId64 "ms,%" PRId64 "\n", end100mS - start100mS, end100mSCt - start100mSCt);
taosHashCleanup(hash1h);
taosHashCleanup(hash1s);
taosHashCleanup(hash10s);
taosHashCleanup(hash100s);
taosHashCleanup(hash1m);
taosHashCleanup(hash10m);
taosHashCleanup(hash100m);
SHashObj *mhash[1000] = {0};
for (int64_t i = 0; i < 1000; ++i) {
mhash[i] = (SHashObj*) taosHashInit(100000, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BINARY), true, HASH_NO_LOCK);
}
for (int64_t i = 0; i < 50000000; ++i) {
#if 0
taosHashPut(mhash[i%1000], name + i * 9, 9, &i, sizeof(i));
#else
taosHashPut(mhash[i/50000], name + i * 9, 9, &i, sizeof(i));
#endif
}
int64_t startMhashCt = 0;
for (int64_t i = 0; i < 1000; ++i) {
startMhashCt += taosHashGetCompTimes(mhash[i]);
}
int64_t startMhash = taosGetTimestampMs();
#if 0
for (int32_t i = 0; i < 10000000; ++i) {
ASSERT(taosHashGet(mhash[i%1000], name + i * 9, 9));
}
#else
// for (int64_t i = 0; i < 10000000; ++i) {
for (int64_t i = 0; i < 50000000; i+=5) {
ASSERT(taosHashGet(mhash[i/50000], name + i * 9, 9));
}
#endif
int64_t endMhash = taosGetTimestampMs();
int64_t endMhashCt = 0;
for (int64_t i = 0; i < 1000; ++i) {
printf(" %" PRId64 , taosHashGetCompTimes(mhash[i]));
endMhashCt += taosHashGetCompTimes(mhash[i]);
}
printf("\n100m \t %" PRId64 "ms,%" PRId64 "\n", endMhash - startMhash, endMhashCt - startMhashCt);
for (int64_t i = 0; i < 1000; ++i) {
taosHashCleanup(mhash[i]);
}
}
}
int main(int argc, char** argv) {
......@@ -210,4 +405,5 @@ TEST(testCase, hashTest) {
noLockPerformanceTest();
multithreadsTest();
acquireRleaseTest();
//perfTest();
}
......@@ -4,6 +4,7 @@ system sh/cfg.sh -n dnode1 -c monitorfqdn -v localhost
system sh/cfg.sh -n dnode1 -c monitorport -v 80
system sh/cfg.sh -n dnode1 -c monitorInterval -v 1
system sh/cfg.sh -n dnode1 -c monitorComp -v 1
system sh/cfg.sh -n dnode1 -c uptimeInterval -v 3
#system sh/cfg.sh -n dnode1 -c supportVnodes -v 128
#system sh/cfg.sh -n dnode1 -c telemetryReporting -v 1
......@@ -14,7 +15,7 @@ system sh/cfg.sh -n dnode1 -c monitorComp -v 1
system sh/exec.sh -n dnode1 -s start
sql connect
print =============== select * from information_schema.ins_dnodes
print =============== create database
sql create database db vgroups 2;
sql use db;
sql create table db.stb (ts timestamp, c1 int, c2 binary(4)) tags(t1 int, t2 binary(16)) comment "abd";
......
......@@ -8,7 +8,20 @@ sql create user sysinfo0 pass 'taosdata'
sql create user sysinfo1 pass 'taosdata'
sql alter user sysinfo0 sysinfo 0
sql alter user sysinfo1 sysinfo 1
sql create database db
sql use db
sql create table db.stb (ts timestamp, i int) tags (t int)
sql create table db.ctb using db.stb tags (1)
sql create table db.ntb (ts timestamp, i int)
sql insert into db.ctb values (now, 1);
sql insert into db.ntb values (now, 1);
sql select * from db.stb
sql select * from db.ctb
sql select * from db.ntb
sql create database d2
sql GRANT all ON d2.* to sysinfo0;
print user sysinfo0 login
sql close
......@@ -17,11 +30,31 @@ sql connect sysinfo0
print =============== check oper
sql_error create user u1 pass 'u1'
sql_error drop user sysinfo1
sql_error alter user sysinfo1 pass '1'
sql_error alter user sysinfo0 pass '1'
sql_error alter user sysinfo0 enable 0
sql_error alter user sysinfo0 enable 1
sql_error alter user sysinfo1 pass '1'
sql_error alter user sysinfo1 enable 1
sql_error alter user sysinfo1 enable 1
sql_error GRANT read ON db.* to sysinfo0;
sql_error GRANT read ON *.* to sysinfo0;
sql_error REVOKE read ON db.* from sysinfo0;
sql_error REVOKE read ON *.* from sysinfo0;
sql_error GRANT write ON db.* to sysinfo0;
sql_error GRANT write ON *.* to sysinfo0;
sql_error REVOKE write ON db.* from sysinfo0;
sql_error REVOKE write ON *.* from sysinfo0;
sql_error REVOKE write ON *.* from sysinfo0;
sql_error create dnode $hostname port 7200
sql_error drop dnode 1
sql_error alter dnode 1 'debugFlag 135'
sql_error alter dnode 1 'dDebugFlag 131'
sql_error alter dnode 1 'resetlog'
sql_error alter dnode 1 'monitor' '1'
sql_error alter dnode 1 'monitor' '0'
sql_error alter dnode 1 'monitor 1'
sql_error alter dnode 1 'monitor 0'
sql_error create qnode on dnode 1
sql_error drop qnode on dnode 1
......@@ -44,20 +77,107 @@ sql_error create database d1
sql_error drop database db
sql_error use db
sql_error alter database db replica 1;
sql_error alter database db keep 21
sql_error show db.vgroups
sql select * from information_schema.ins_stables where db_name = 'db'
sql select * from information_schema.ins_tables where db_name = 'db'
sql_error create table db.stb1 (ts timestamp, i int) tags (t int)
sql_error create table db.ctb1 using db.stb1 tags (1)
sql_error create table db.ntb1 (ts timestamp, i int)
sql_error insert into db.ctb values (now, 1);
sql_error insert into db.ntb values (now, 1);
sql_error select * from db.stb
sql_error select * from db.ctb
sql_error select * from db.ntb
sql use d2
sql create table d2.stb2 (ts timestamp, i int) tags (t int)
sql create table d2.ctb2 using d2.stb2 tags (1)
sql create table d2.ntb2 (ts timestamp, i int)
sql insert into d2.ctb2 values (now, 1);
sql insert into d2.ntb2 values (now, 1);
sql select * from d2.stb2
sql select * from d2.ctb2
sql select * from d2.ntb2
print =============== check show
sql select * from information_schema.ins_users
sql_error show users
sql_error show cluster
sql_error select * from information_schema.ins_dnodes
sql_error select * from information_schema.ins_mnodes
sql_error show snodes
sql_error select * from information_schema.ins_qnodes
sql_error show dnodes
sql_error show snodes
sql_error show qnodes
sql_error show mnodes
sql_error show bnodes
sql_error show db.vgroups
sql_error show db.stables
sql_error show db.tables
sql_error show indexes from stb from db
sql show databases
sql_error show d2.vgroups
sql show d2.stables
sql show d2.tables
sql show indexes from stb2 from d2
#sql_error show create database db
sql_error show create table db.stb;
sql_error show create table db.ctb;
sql_error show create table db.ntb;
sql show streams
sql show consumers
sql show topics
sql show subscriptions
sql show functions
sql_error show grants
sql show queries
sql show connections
sql show apps
sql_error show transactions
#sql_error show create database d2
sql show create table d2.stb2;
sql show create table d2.ctb2;
sql show create table d2.ntb2;
sql_error show variables;
sql show local variables;
sql_error show dnode 1 variables;
sql_error show variables;
system sh/exec.sh -n dnode1 -s stop -x SIGINT
print =============== check information_schema
sql show databases
if $rows != 3 then
return -1
endi
sql use information_schema;
sql_error select * from information_schema.ins_dnodes
sql_error select * from information_schema.ins_mnodes
sql_error select * from information_schema.ins_modules
sql_error select * from information_schema.ins_qnodes
sql_error select * from information_schema.ins_cluster
sql select * from information_schema.ins_databases
sql select * from information_schema.ins_functions
sql select * from information_schema.ins_indexes
sql select * from information_schema.ins_stables
sql select * from information_schema.ins_tables
sql select * from information_schema.ins_tags
sql select * from information_schema.ins_users
sql_error select * from information_schema.ins_grants
sql_error select * from information_schema.ins_vgroups
sql_error select * from information_schema.ins_configs
sql_error select * from information_schema.ins_dnode_variables
print =============== check performance_schema
sql use performance_schema;
sql select * from performance_schema.perf_connections
sql select * from performance_schema.perf_queries
sql select * from performance_schema.perf_topics
sql select * from performance_schema.perf_consumers
sql select * from performance_schema.perf_subscriptions
#sql_error select * from performance_schema.perf_trans
#sql_error select * from performance_schema.perf_smas
#sql_error select * from information_schema.perf_streams
#sql_error select * from information_schema.perf_apps
#system sh/exec.sh -n dnode1 -s stop -x SIGINT
Markdown is supported
0% .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑!
想要评论请 注册