Skip to content

T
TDengine

taosdata / TDengine
大约 2 年 前同步成功

通知 1192
Star 22018
Fork 4786
T
TDengine
未验证 提交 b924ff08 编写于 作者: S Shengliang Guan 提交者： GitHub
浏览文件
操作

Merge pull request #16022 from taosdata/fix/release 

release: merge from 3.0 to main
上级 5edce328 2040ab78
展开全部 隐藏空白更改
内联 并排
Showing with 1711 addition and 1286 deletion
Jenkinsfile2
浏览文件 @ b924ff08
......@@ -265,6 +265,7 @@ def pre_test_build_win() {
'''
bat '''
cd %WIN_CONNECTOR_ROOT%
python.exe -m pip install --upgrade pip
python -m pip install .
xcopy /e/y/i/f %WIN_INTERNAL_ROOT%\\debug\\build\\lib\\taos.dll C:\\Windows\\System32
'''
......@@ -283,6 +284,7 @@ def run_win_test() {
bat '''
echo "windows test ..."
cd %WIN_CONNECTOR_ROOT%
python.exe -m pip install --upgrade pip
python -m pip install .
xcopy /e/y/i/f %WIN_INTERNAL_ROOT%\\debug\\build\\lib\\taos.dll C:\\Windows\\System32
ls -l C:\\Windows\\System32\\taos.dll
......
README-CN.md
浏览文件 @ b924ff08
......@@ -14,7 +14,6 @@
[![Build status](https://ci.appveyor.com/api/projects/status/kf3pwh2or5afsgl9/branch/master?svg=true)](https://ci.appveyor.com/project/sangshuduo/tdengine-2n8ge/branch/master)
[![Coverage Status](https://coveralls.io/repos/github/taosdata/TDengine/badge.svg?branch=develop)](https://coveralls.io/github/taosdata/TDengine?branch=develop)
[![CII Best Practices](https://bestpractices.coreinfrastructure.org/projects/4201/badge)](https://bestpractices.coreinfrastructure.org/projects/4201)
[![tdengine](https://snapcraft.io//tdengine/badge.svg)](https://snapcraft.io/tdengine)
简体中文 | [English](README.md) | 很多职位正在热招中，请看[这里](https://www.taosdata.com/cn/careers/)
......@@ -36,11 +35,13 @@ TDengine 是一款开源、高性能、云原生的时序数据库 (Time-Series
# 文档
关于完整的使用手册，系统架构和更多细节，请参考 [TDengine 文档](https://docs.taosdata.com) 或者 [English Version](https://docs.tdengine.com)。
关于完整的使用手册，系统架构和更多细节，请参考 [TDengine 文档](https://docs.taosdata.com) 或者 [English Documents](https://docs.tdengine.com)。
# 构建
TDengine 目前 2.0 版服务器仅能在 Linux 系统上安装和运行，后续会支持 Windows、macOS 等系统。客户端可以在 Windows 或 Linux 上安装和运行。任何 OS 的应用也可以选择 RESTful 接口连接服务器 taosd。CPU 支持 X64/ARM64/MIPS64/Alpha64，后续会支持 ARM32、RISC-V 等 CPU 架构。用户可根据需求选择通过源码或者[安装包](https://docs.taosdata.com/get-started/package/)来安装。本快速指南仅适用于通过源码安装。
TDengine 目前可以在 Linux、 Windows 等平台上安装和运行。任何 OS 的应用也可以选择 taosAdapter 的 RESTful 接口连接服务端 taosd。CPU 支持 X64/ARM64，后续会支持 MIPS64、Alpha64、ARM32、RISC-V 等 CPU 架构。
用户可根据需求选择通过[源码](https://www.taosdata.com/cn/getting-started/#通过源码安装)或者[安装包](https://www.taosdata.com/cn/getting-started/#通过安装包安装)来安装。本快速指南仅适用于通过源码安装。
## 安装工具
......@@ -50,20 +51,6 @@ TDengine 目前 2.0 版服务器仅能在 Linux 系统上安装和运行，后
sudo apt-get install -y gcc cmake build-essential git libssl-dev
```
编译或打包 JDBC 驱动源码，需安装 Java JDK 8 或以上版本和 Apache Maven 2.7 或以上版本。
安装 OpenJDK 8：
```bash
sudo apt-get install -y openjdk-8-jdk
```
安装 Apache Maven：
```bash
sudo apt-get install -y maven
```
#### 为 taos-tools 安装编译需要的软件
taosTools 是用于 TDengine 的辅助工具软件集合。目前它包含 taosBenchmark（曾命名为 taosdemo）和 taosdump 两个软件。
......@@ -79,19 +66,10 @@ sudo apt install build-essential libjansson-dev libsnappy-dev liblzma-dev libz-d
### CentOS 7.9：
```bash
sudo yum install -y gcc gcc-c++ make cmake git openssl-devel
```
安装 OpenJDK 8：
```bash
sudo yum install -y java-1.8.0-openjdk
```
安装 Apache Maven：
```bash
sudo yum install -y maven
sudo yum install epel-release
sudo yum update
sudo yum install -y gcc gcc-c++ make cmake3 git openssl-devel
sudo ln -sf /usr/bin/cmake3 /usr/bin/cmake
```
### CentOS 8 & Fedora
......@@ -100,29 +78,29 @@ sudo yum install -y maven
sudo dnf install -y gcc gcc-c++ make cmake epel-release git openssl-devel
```
安装 OpenJDK 8：
#### 在 CentOS 上构建 taosTools 安装依赖软件
#### For CentOS 7/RHEL
```bash
sudo dnf install -y java-1.8.0-openjdk
```
sudo yum install -y zlib-devel xz-devel snappy-devel jansson jansson-devel pkgconfig libatomic libstdc++-static openssl-devel
```
安装 Apache Maven：
#### For CentOS 8/Rocky Linux
```bash
sudo dnf install -y maven
```
sudo yum install -y epel-release
sudo yum install -y dnf-plugins-core
sudo yum config-manager --set-enabled powertools
sudo yum install -y zlib-devel xz-devel snappy-devel jansson jansson-devel pkgconfig libatomic libstdc++-static openssl-devel
```
#### 在 CentOS 上构建 taosTools 安装依赖软件
为了在 CentOS 上构建 [taosTools](https://github.com/taosdata/taos-tools) 需要安装如下依赖软件
注意：由于 snappy 缺乏 pkg-config 支持（参考 [链接](https://github.com/google/snappy/pull/86)），会导致 cmake 提示无法发现 libsnappy，实际上工作正常。
```bash
sudo yum install zlib-devel xz-devel snappy-devel jansson jansson-devel pkgconfig libatomic libstdc++-static openssl-devel
若 powertools 安装失败，可以尝试改用：
```
sudo yum config-manager --set-enabled Powertools
```
注意：由于 snappy 缺乏 pkg-config 支持
（参考 [链接](https://github.com/google/snappy/pull/86)），会导致
cmake 提示无法发现 libsnappy，实际上工作正常。
### 设置 golang 开发环境
......@@ -135,6 +113,12 @@ go env -w GO111MODULE=on
go env -w GOPROXY=https://goproxy.cn,direct
```
默认情况下，内嵌的 http 服务仍然可以从 TDengine 源码构建。当然您也可以使用以下命令选择构建 taosAdapter 作为 RESTful 接口的服务。
```
cmake .. -DBUILD_HTTP=false
```
### 设置 rust 开发环境
TDengine 包含数个使用 Rust 语言开发的组件. 请参考 rust-lang.org 官方文档设置 rust 开发环境。
......@@ -275,24 +259,6 @@ nmake install
sudo make install
```
安装成功后，如果想以服务形式启动，先配置 `.plist` 文件，在终端中执行：
```bash
sudo cp ../packaging/macOS/com.taosdata.tdengine.plist /Library/LaunchDaemons
```
在终端中启动 TDengine 服务：
```bash
sudo launchctl load /Library/LaunchDaemons/com.taosdata.tdengine.plist
```
在终端中停止 TDengine 服务：
```bash
sudo launchctl unload /Library/LaunchDaemons/com.taosdata.tdengine.plist
```
## 快速运行
如果不希望以服务方式运行 TDengine，也可以在终端中直接运行它。也即在生成完成后，执行以下命令（在 Windows 下，生成的可执行文件会带有 .exe 后缀，例如会名为 taosd.exe ）：
......@@ -342,21 +308,6 @@ TDengine 提供了丰富的应用程序开发接口，其中包括 C/C++、Java
- [C#](https://docs.taosdata.com/reference/connector/csharp/)
- [RESTful API](https://docs.taosdata.com/reference/rest-api/)
## 第三方连接器
TDengine 社区生态中也有一些非常友好的第三方连接器，可以通过以下链接访问它们的源码。
- [Rust Bindings](https://github.com/songtianyi/tdengine-rust-bindings/tree/master/examples)
- [.Net Core Connector](https://github.com/maikebing/Maikebing.EntityFrameworkCore.Taos)
- [Lua Connector](https://github.com/taosdata/TDengine/tree/develop/examples/lua)
- [PHP](https://www.taosdata.com/en/documentation/connector#c-cpp)
# 运行和添加测试例
TDengine 的测试框架和所有测试例全部开源。
点击 [这里](https://github.com/taosdata/TDengine/blob/develop/tests/How-To-Run-Test-And-How-To-Add-New-Test-Case.md)，了解如何运行测试例和添加新的测试例。
# 成为社区贡献者
点击 [这里](https://www.taosdata.com/cn/contributor/)，了解如何成为 TDengine 的贡献者。
......@@ -364,7 +315,3 @@ TDengine 的测试框架和所有测试例全部开源。
# 加入技术交流群
TDengine 官方社群「物联网大数据群」对外开放，欢迎您加入讨论。搜索微信号 "tdengine"，加小 T 为好友，即可入群。
# [谁在使用 TDengine](https://github.com/taosdata/TDengine/issues/2432)
欢迎所有 TDengine 用户及贡献者在 [这里](https://github.com/taosdata/TDengine/issues/2432) 分享您在当前工作中开发/使用 TDengine 的故事。
README.md
浏览文件 @ b924ff08
......@@ -20,7 +20,7 @@ English | [简体中文](README-CN.md) | We are hiring, check [here](https://tde
# What is TDengine？
TDengine is an open source, cloud native time-series database optimized for Internet of Things (IoT), Connected Cars, and Industrial IoT. It enables efficient, real-time data ingestion, processing, and monitoring of TB and even PB scale data per day, generated by billions of sensors and data collectors. Below are the most outstanding advantages of TDengine:
TDengine is an open source, cloud native time-series database optimized for Internet of Things (IoT), Connected Cars, and Industrial IoT. It enables efficient, real-time data ingestion, processing, and monitoring of TB and even PB scale data per day, generated by billions of sensors and data collectors. TDengine differentiates itself from other TSDBs with the following advantages.:
- High-Performance: TDengine is the only time-series database to solve the high cardinality issue to support billions of data collection points while out performing other time-series databases for data ingestion, querying and data compression.
......@@ -28,7 +28,7 @@ TDengine is an open source, cloud native time-series database optimized for Inte
- Cloud Native: Through native distributed design, sharding and partitioning, separation of compute and storage, RAFT, support for kubernetes deployment and full observability, TDengine can be deployed on public, private or hybrid clouds.
- Open Source: TDengine’s core modules, including cluster feature, are all available under open source licenses. It has gathered 18.7k stars on GitHub, an active developer community, and over 137k running instances worldwide.
- Open Source: TDengine’s core modules, including cluster feature, are all available under open source licenses. It has gathered 18.8k stars on GitHub, an active developer community, and over 137k running instances worldwide.
- Ease of Use: For administrators, TDengine significantly reduces the effort to deploy and maintain. For developers, it provides a simple interface, simplified solution and seamless integrations for third party tools. For data users, it gives easy data access.
......@@ -52,20 +52,6 @@ To build TDengine, use [CMake](https://cmake.org/) 3.0.2 or higher versions in t
sudo apt-get install -y gcc cmake build-essential git libssl-dev
```
To compile and package the JDBC driver source code, you should have a Java jdk-8 or higher and Apache Maven 2.7 or higher installed.
To install openjdk-8:
```bash
sudo apt-get install -y openjdk-8-jdk
```
To install Apache Maven:
```bash
sudo apt-get install -y maven
```
#### Install build dependencies for taosTools
We provide a few useful tools such as taosBenchmark (was named taosdemo) and taosdump. They were part of TDengine. From TDengine 2.4.0.0, taosBenchmark and taosdump were not released together with TDengine.
......@@ -86,36 +72,12 @@ sudo yum install -y gcc gcc-c++ make cmake3 git openssl-devel
sudo ln -sf /usr/bin/cmake3 /usr/bin/cmake
```
To install openjdk-8:
```bash
sudo yum install -y java-1.8.0-openjdk
```
To install Apache Maven:
```bash
sudo yum install -y maven
```
### CentOS 8 & Fedora
```bash
sudo dnf install -y gcc gcc-c++ make cmake epel-release git openssl-devel
```
To install openjdk-8:
```bash
sudo dnf install -y java-1.8.0-openjdk
```
To install Apache Maven:
```bash
sudo dnf install -y maven
```
#### Install build dependencies for taosTools on CentOS
To build the [taosTools](https://github.com/taosdata/taos-tools) on CentOS, the following packages need to be installed.
......@@ -124,11 +86,11 @@ To build the [taosTools](https://github.com/taosdata/taos-tools) on CentOS, the
sudo yum install zlib-devel xz-devel snappy-devel jansson jansson-devel pkgconfig libatomic libstdc++-static openssl-devel
```
Note: Since snappy lacks pkg-config support (refer to [link](https://github.com/google/snappy/pull/86)), it lead a cmake prompt libsnappy not found. But snappy will works well.
Note: Since snappy lacks pkg-config support (refer to [link](https://github.com/google/snappy/pull/86)), it leads a cmake prompt libsnappy not found. But snappy will works well.
### Setup golang environment
TDengine includes few components developed by Go language. Please refer to golang.org official documentation for golang environment setup.
TDengine includes a few components developed by Go language. Please refer to golang.org official documentation for golang environment setup.
Please use version 1.14+. For the user in China, we recommend using a proxy to accelerate package downloading.
......@@ -139,7 +101,7 @@ go env -w GOPROXY=https://goproxy.cn,direct
### Setup rust environment
TDengine includees few compoments developed by Rust language. Please refer to rust-lang.org official documentation for rust environment setup.
TDengine includees a few compoments developed by Rust language. Please refer to rust-lang.org official documentation for rust environment setup.
## Get the source codes
......@@ -304,24 +266,6 @@ After building successfully, TDengine can be installed by:
sudo make install
```
To start the service after installation, config `.plist` file first, in a terminal, use:
```bash
sudo cp ../packaging/macOS/com.taosdata.tdengine.plist /Library/LaunchDaemons
```
To start the service, in a terminal, use:
```bash
sudo launchctl load /Library/LaunchDaemons/com.taosdata.tdengine.plist
```
To stop the service, in a terminal, use:
```bash
sudo launchctl unload /Library/LaunchDaemons/com.taosdata.tdengine.plist
```
## Quick Run
If you don't want to run TDengine as a service, you can run it in current shell. For example, to quickly start a TDengine server after building, run the command below in terminal: (We take Linux as an example, command on Windows will be `taosd.exe`)
......@@ -371,15 +315,6 @@ TDengine provides abundant developing tools for users to develop on TDengine. Fo
- [C#](https://docs.taosdata.com/reference/connector/csharp/)
- [RESTful API](https://docs.taosdata.com/reference/rest-api/)
## Third Party Connectors
The TDengine community has also kindly built some of their own connectors! Follow the links below to find the source code for them.
- [Rust Bindings](https://github.com/songtianyi/tdengine-rust-bindings/tree/master/examples)
- [.Net Core Connector](https://github.com/maikebing/Maikebing.EntityFrameworkCore.Taos)
- [Lua Connector](https://github.com/taosdata/TDengine/tree/develop/tests/examples/lua)
- [PHP](https://www.taosdata.com/en/documentation/connector#c-cpp)
# How to run the test cases and how to add a new test case
TDengine's test framework and all test cases are fully open source.
......
cmake/cmake.install
浏览文件 @ b924ff08
IF (TD_LINUX)
IF (EXISTS /var/lib/taos/dnode/dnodeCfg.json)
INSTALL(CODE "MESSAGE(\"The default data directory /var/lib/taos contains old data of tdengine 2.x, please clear it before installing!\")")
ELSEIF (EXISTS C:/TDengine/data/dnode/dnodeCfg.json)
INSTALL(CODE "MESSAGE(\"The default data directory C:/TDengine/data contains old data of tdengine 2.x, please clear it before installing!\")")
ELSEIF (TD_LINUX)
SET(TD_MAKE_INSTALL_SH "${TD_SOURCE_DIR}/packaging/tools/make_install.sh")
INSTALL(CODE "MESSAGE(\"make install script: ${TD_MAKE_INSTALL_SH}\")")
INSTALL(CODE "execute_process(COMMAND bash ${TD_MAKE_INSTALL_SH} ${TD_SOURCE_DIR} ${PROJECT_BINARY_DIR} Linux ${TD_VER_NUMBER})")
......
cmake/taostools_CMakeLists.txt.in
浏览文件 @ b924ff08
......@@ -2,7 +2,7 @@
# taos-tools
ExternalProject_Add(taos-tools
GIT_REPOSITORY https://github.com/taosdata/taos-tools.git
GIT_TAG 79bf23d
GIT_TAG 43924b8
SOURCE_DIR "${TD_SOURCE_DIR}/tools/taos-tools"
BINARY_DIR ""
#BUILD_IN_SOURCE TRUE
......
docs/en/07-develop/03-insert-data/02-influxdb-line.mdx
浏览文件 @ b924ff08
......@@ -55,9 +55,6 @@ For more details please refer to [InfluxDB Line Protocol](https://docs.influxdat
<TabItem label="Go" value="go">
<GoLine />
</TabItem>
<TabItem label="Rust" value="rust">
<RustLine />
</TabItem>
<TabItem label="Node.js" value="nodejs">
<NodeLine />
</TabItem>
......
docs/en/07-develop/03-insert-data/03-opentsdb-telnet.mdx
浏览文件 @ b924ff08
......@@ -46,9 +46,6 @@ Please refer to [OpenTSDB Telnet API](http://opentsdb.net/docs/build/html/api_te
<TabItem label="Go" value="go">
<GoTelnet />
</TabItem>
<TabItem label="Rust" value="rust">
<RustTelnet />
</TabItem>
<TabItem label="Node.js" value="nodejs">
<NodeTelnet />
</TabItem>
......
docs/en/07-develop/03-insert-data/04-opentsdb-json.mdx
浏览文件 @ b924ff08
......@@ -63,9 +63,6 @@ Please refer to [OpenTSDB HTTP API](http://opentsdb.net/docs/build/html/api_http
<TabItem label="Go" value="go">
<GoJson />
</TabItem>
<TabItem label="Rust" value="rust">
<RustJson />
</TabItem>
<TabItem label="Node.js" value="nodejs">
<NodeJson />
</TabItem>
......
docs/en/07-develop/03-insert-data/_rust_line.mdx
浏览文件 @ b924ff08
```rust
{{#include docs/examples/rust/schemalessexample/examples/influxdb_line_example.rs}}
```
docs/en/07-develop/03-insert-data/_rust_opts_json.mdx
浏览文件 @ b924ff08
```rust
{{#include docs/examples/rust/schemalessexample/examples/opentsdb_json_example.rs}}
```
docs/en/07-develop/03-insert-data/_rust_opts_telnet.mdx
浏览文件 @ b924ff08
```rust
{{#include docs/examples/rust/schemalessexample/examples/opentsdb_telnet_example.rs}}
```
docs/en/14-reference/13-schemaless/13-schemaless.md
浏览文件 @ b924ff08
......@@ -3,7 +3,7 @@ title: Schemaless Writing
description: "The Schemaless write method eliminates the need to create super tables/sub tables in advance and automatically creates the storage structure corresponding to the data, as it is written to the interface."
---
In IoT applications, data is collected for many purposes such as intelligent control, business analysis, device monitoring and so on. Due to changes in business or functional requirements or changes in device hardware, the application logic and even the data collected may change. To provide the flexibility needed in such cases and in a rapidly changing IoT landscape, TDengine starting from version 2.2.0.0, provides a series of interfaces for the schemaless writing method. These interfaces eliminate the need to create super tables and subtables in advance by automatically creating the storage structure corresponding to the data as the data is written to the interface. When necessary, schemaless writing will automatically add the required columns to ensure that the data written by the user is stored correctly.
In IoT applications, data is collected for many purposes such as intelligent control, business analysis, device monitoring and so on. Due to changes in business or functional requirements or changes in device hardware, the application logic and even the data collected may change. To provide the flexibility needed in such cases and in a rapidly changing IoT landscape, TDengine provides a series of interfaces for the schemaless writing method. These interfaces eliminate the need to create super tables and subtables in advance by automatically creating the storage structure corresponding to the data as the data is written to the interface. When necessary, schemaless writing will automatically add the required columns to ensure that the data written by the user is stored correctly.
The schemaless writing method creates super tables and their corresponding subtables. These are completely indistinguishable from the super tables and subtables created directly via SQL. You can write data directly to them via SQL statements. Note that the names of tables created by schemaless writing are based on fixed mapping rules for tag values, so they are not explicitly ideographic and they lack readability.
......@@ -39,10 +39,10 @@ In the schemaless writing data line protocol, each data item in the field_set ne
| -------- | -------- | ------------ | -------------- |
| 1 | none or f64 | double | 8 |
| 2 | f32 | float | 4 |
| 3 | i8 | TinyInt | 1 |
| 4 | i16 | SmallInt | 2 |
| 5 | i32 | Int | 4 |
| 6 | i64 or i | Bigint | 8 |
| 3 | i8/u8 | TinyInt/UTinyInt | 1 |
| 4 | i16/u16 | SmallInt/USmallInt | 2 |
| 5 | i32/u32 | Int/UInt | 4 |
| 6 | i64/i/u64/u | Bigint/Bigint/UBigint/UBigint | 8 |
- `t`, `T`, `true`, `True`, `TRUE`, `f`, `F`, `false`, and `False` will be handled directly as BOOL types.
......@@ -72,11 +72,11 @@ If the subtable obtained by the parse line protocol does not exist, Schemaless c
4. If the specified tag or regular column in the data row does not exist, the corresponding tag or regular column is added to the super table (only incremental).
5. If there are some tag columns or regular columns in the super table that are not specified to take values in a data row, then the values of these columns are set to NULL.
6. For BINARY or NCHAR columns, if the length of the value provided in a data row exceeds the column type limit, the maximum length of characters allowed to be stored in the column is automatically increased (only incremented and not decremented) to ensure complete preservation of the data.
7. If the specified data subtable already exists, and the specified tag column takes a value different from the saved value this time, the value in the latest data row overwrites the old tag column take value.
8. Errors encountered throughout the processing will interrupt the writing process and return an error code.
7. Errors encountered throughout the processing will interrupt the writing process and return an error code.
8. In order to improve the efficiency of writing, it is assumed by default that the order of the fields in the same Super is the same (the first data contains all fields, and the following data is in this order). If the order is different, the parameter smlDataFormat needs to be configured to be false. Otherwise, the data is written in the same order, and the data in the library will be abnormal.
:::tip
All processing logic of schemaless will still follow TDengine's underlying restrictions on data structures, such as the total length of each row of data cannot exceed 48k bytes. See [TAOS SQL Boundary Limits](/taos-sql/limit) for specific constraints in this area.
All processing logic of schemaless will still follow TDengine's underlying restrictions on data structures, such as the total length of each row of data cannot exceed 16k bytes. See [TAOS SQL Boundary Limits](/taos-sql/limit) for specific constraints in this area.
:::
## Time resolution recognition
......
docs/examples/rust/Cargo.toml
浏览文件 @ b924ff08
[workspace]
members = ["restexample", "nativeexample", "schemalessexample"]
members = ["restexample", "nativeexample"]
docs/examples/rust/nativeexample/Cargo.toml
浏览文件 @ b924ff08
......@@ -5,6 +5,9 @@ edition = "2021"
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[dependencies]
libtaos = { version = "0.4.3" }
tokio = { version = "*", features = ["rt", "macros", "rt-multi-thread"] }
bstr = { version = "*" }
anyhow = "1"
chrono = "0.4"
serde = { version = "1", features = ["derive"] }
tokio = { version = "1", features = ["rt", "macros", "rt-multi-thread"] }
taos = { version = "0.*" }
docs/examples/rust/nativeexample/examples/connect.rs
浏览文件 @ b924ff08
use libtaos::*;
use taos::*;
fn taos_connect() -> Result<Taos, Error> {
TaosCfgBuilder::default()
.ip("localhost")
.user("root")
.pass("taosdata")
// .db("log") // remove comment if you want to connect to database log by default.
.port(6030u16)
.build()
.expect("TaosCfg builder error")
.connect()
}
fn main() {
#[tokio::main]
async fn main() -> Result<(), Error> {
#[allow(unused_variables)]
let taos = taos_connect().unwrap();
println!("Connected")
let taos = TaosBuilder::from_dsn("taos://")?.build()?;
println!("Connected");
Ok(())
}
docs/examples/rust/nativeexample/examples/stmt_example.rs
浏览文件 @ b924ff08
use bstr::BString;
use libtaos::*;
use taos::*;
#[tokio::main]
async fn main() -> Result<(), Error> {
let taos = TaosCfg::default().connect().expect("fail to connect");
async fn main() -> anyhow::Result<()> {
let taos = TaosBuilder::from_dsn("taos://")?.build()?;
taos.create_database("power").await?;
taos.use_database("power").await?;
taos.exec("CREATE STABLE meters (ts TIMESTAMP, current FLOAT, voltage INT, phase FLOAT) TAGS (location BINARY(64), groupId INT)").await?;
let mut stmt = taos.stmt("INSERT INTO ? USING meters TAGS(?, ?) VALUES(?, ?, ?, ?)")?;
taos.exec("CREATE STABLE IF NOT EXISTS meters (ts TIMESTAMP, current FLOAT, voltage INT, phase FLOAT) TAGS (location BINARY(64), groupId INT)").await?;
let mut stmt = Stmt::init(&taos)?;
stmt.prepare("INSERT INTO ? USING meters TAGS(?, ?) VALUES(?, ?, ?, ?)")?;
// bind table name and tags
stmt.set_tbname_tags(
"d1001",
[
Field::Binary(BString::from("California.SanFrancisco")),
Field::Int(2),
],
&[Value::VarChar("San Fransico".into()), Value::Int(2)],
)?;
// bind values.
let values = vec![
Field::Timestamp(Timestamp::new(1648432611249, TimestampPrecision::Milli)),
Field::Float(10.3),
Field::Int(219),
Field::Float(0.31),
ColumnView::from_millis_timestamp(vec![1648432611249]),
ColumnView::from_floats(vec![10.3]),
ColumnView::from_ints(vec![219]),
ColumnView::from_floats(vec![0.31]),
];
stmt.bind(&values)?;
// bind one more row
let values2 = vec![
Field::Timestamp(Timestamp::new(1648432611749, TimestampPrecision::Milli)),
Field::Float(12.6),
Field::Int(218),
Field::Float(0.33),
ColumnView::from_millis_timestamp(vec![1648432611749]),
ColumnView::from_floats(vec![12.6]),
ColumnView::from_ints(vec![218]),
ColumnView::from_floats(vec![0.33]),
];
stmt.bind(&values2)?;
// execute
stmt.execute()?;
stmt.add_batch()?;
// execute.
let rows = stmt.execute()?;
assert_eq!(rows, 2);
Ok(())
}
docs/examples/rust/nativeexample/examples/subscribe_demo.rs
浏览文件 @ b924ff08
fn main() {
}
\ No newline at end of file
use std::time::Duration;
use chrono::{DateTime, Local};
use taos::*;
// Query options 2, use deserialization with serde.
#[derive(Debug, serde::Deserialize)]
#[allow(dead_code)]
struct Record {
// deserialize timestamp to chrono::DateTime<Local>
ts: DateTime<Local>,
// float to f32
current: Option<f32>,
// int to i32
voltage: Option<i32>,
phase: Option<f32>,
}
async fn prepare(taos: Taos) -> anyhow::Result<()> {
let inserted = taos.exec_many([
// create child table
"CREATE TABLE `d0` USING `meters` TAGS(0, 'Los Angles')",
// insert into child table
"INSERT INTO `d0` values(now - 10s, 10, 116, 0.32)",
// insert with NULL values
"INSERT INTO `d0` values(now - 8s, NULL, NULL, NULL)",
// insert and automatically create table with tags if not exists
"INSERT INTO `d1` USING `meters` TAGS(1, 'San Francisco') values(now - 9s, 10.1, 119, 0.33)",
// insert many records in a single sql
"INSERT INTO `d1` values (now-8s, 10, 120, 0.33) (now - 6s, 10, 119, 0.34) (now - 4s, 11.2, 118, 0.322)",
]).await?;
assert_eq!(inserted, 6);
Ok(())
}
#[tokio::main]
async fn main() -> anyhow::Result<()> {
let dsn = "taos://localhost:6030";
let builder = TaosBuilder::from_dsn(dsn)?;
let taos = builder.build()?;
let db = "tmq";
// prepare database
taos.exec_many([
format!("DROP TOPIC IF EXISTS tmq_meters"),
format!("DROP DATABASE IF EXISTS `{db}`"),
format!("CREATE DATABASE `{db}`"),
format!("USE `{db}`"),
// create super table
format!("CREATE TABLE `meters` (`ts` TIMESTAMP, `current` FLOAT, `voltage` INT, `phase` FLOAT) TAGS (`groupid` INT, `location` BINARY(16))"),
// create topic for subscription
format!("CREATE TOPIC tmq_meters with META AS DATABASE {db}")
])
.await?;
let task = tokio::spawn(prepare(taos));
tokio::time::sleep(Duration::from_secs(1)).await;
// subscribe
let tmq = TmqBuilder::from_dsn("taos://localhost:6030/?group.id=test")?;
let mut consumer = tmq.build()?;
consumer.subscribe(["tmq_meters"]).await?;
{
let mut stream = consumer.stream();
while let Some((offset, message)) = stream.try_next().await? {
// get information from offset
// the topic
let topic = offset.topic();
// the vgroup id, like partition id in kafka.
let vgroup_id = offset.vgroup_id();
println!("* in vgroup id {vgroup_id} of topic {topic}\n");
if let Some(data) = message.into_data() {
while let Some(block) = data.fetch_raw_block().await? {
// one block for one table, get table name if needed
let name = block.table_name();
let records: Vec<Record> = block.deserialize().try_collect()?;
println!(
"** table: {}, got {} records: {:#?}\n",
name.unwrap(),
records.len(),
records
);
}
}
consumer.commit(offset).await?;
}
}
consumer.unsubscribe().await;
task.await??;
Ok(())
}
docs/examples/rust/restexample/Cargo.toml
浏览文件 @ b924ff08
......@@ -4,5 +4,9 @@ version = "0.1.0"
edition = "2021"
[dependencies]
libtaos = { version = "0.4.3", features = ["rest"] }
tokio = { version = "*", features = ["rt", "macros", "rt-multi-thread"] }
anyhow = "1"
chrono = "0.4"
serde = { version = "1", features = ["derive"] }
tokio = { version = "1", features = ["rt", "macros", "rt-multi-thread"] }
taos = { version = "0.*" }
docs/examples/rust/restexample/examples/connect.rs
浏览文件 @ b924ff08
use libtaos::*;
fn taos_connect() -> Result<Taos, Error> {
TaosCfgBuilder::default()
.ip("localhost")
.user("root")
.pass("taosdata")
// .db("log") // remove comment if you want to connect to database log by default.
.port(6030u16)
.build()
.expect("TaosCfg builder error")
.connect()
}
use taos::*;
#[tokio::main]
async fn main() {
async fn main() -> Result<(), Error> {
#[allow(unused_variables)]
let taos = taos_connect().expect("connect error");
println!("Connected")
let taos = TaosBuilder::from_dsn("taos://")?.build()?;
println!("Connected");
Ok(())
}
docs/examples/rust/restexample/examples/insert_example.rs
浏览文件 @ b924ff08
use libtaos::*;
use taos::*;
#[tokio::main]
async fn main() -> Result<(), Error> {
let taos = TaosCfg::default().connect().expect("fail to connect");
taos.create_database("power").await?;
taos.exec("CREATE STABLE power.meters (ts TIMESTAMP, current FLOAT, voltage INT, phase FLOAT) TAGS (location BINARY(64), groupId INT)").await?;
let sql = "INSERT INTO power.d1001 USING power.meters TAGS(California.SanFrancisco, 2) VALUES ('2018-10-03 14:38:05.000', 10.30000, 219, 0.31000) ('2018-10-03 14:38:15.000', 12.60000, 218, 0.33000) ('2018-10-03 14:38:16.800', 12.30000, 221, 0.31000)
power.d1002 USING power.meters TAGS(California.SanFrancisco, 3) VALUES ('2018-10-03 14:38:16.650', 10.30000, 218, 0.25000)
power.d1003 USING power.meters TAGS(California.LosAngeles, 2) VALUES ('2018-10-03 14:38:05.500', 11.80000, 221, 0.28000) ('2018-10-03 14:38:16.600', 13.40000, 223, 0.29000)
power.d1004 USING power.meters TAGS(California.LosAngeles, 3) VALUES ('2018-10-03 14:38:05.000', 10.80000, 223, 0.29000) ('2018-10-03 14:38:06.500', 11.50000, 221, 0.35000)";
let result = taos.query(sql).await?;
println!("{:?}", result);
async fn main() -> anyhow::Result<()> {
let dsn = "ws://";
let taos = TaosBuilder::from_dsn(dsn)?.build()?;
taos.exec_many([
"DROP DATABASE IF EXISTS power",
"CREATE DATABASE power",
"USE power",
"CREATE STABLE power.meters (ts TIMESTAMP, current FLOAT, voltage INT, phase FLOAT) TAGS (location BINARY(64), groupId INT)"
]).await?;
let inserted = taos.exec("INSERT INTO
power.d1001 USING power.meters TAGS('San Francisco', 2)
VALUES ('2018-10-03 14:38:05.000', 10.30000, 219, 0.31000)
('2018-10-03 14:38:15.000', 12.60000, 218, 0.33000) ('2018-10-03 14:38:16.800', 12.30000, 221, 0.31000)
power.d1002 USING power.meters TAGS('San Francisco', 3)
VALUES ('2018-10-03 14:38:16.650', 10.30000, 218, 0.25000)
power.d1003 USING power.meters TAGS('Los Angeles', 2)
VALUES ('2018-10-03 14:38:05.500', 11.80000, 221, 0.28000) ('2018-10-03 14:38:16.600', 13.40000, 223, 0.29000)
power.d1004 USING power.meters TAGS('Los Angeles', 3)
VALUES ('2018-10-03 14:38:05.000', 10.80000, 223, 0.29000) ('2018-10-03 14:38:06.500', 11.50000, 221, 0.35000)").await?;
assert_eq!(inserted, 8);
Ok(())
}
// output:
// TaosQueryData { column_meta: [ColumnMeta { name: "affected_rows", type_: Int, bytes: 4 }], rows: [[Int(8)]] }
docs/examples/rust/restexample/examples/query_example.rs
浏览文件 @ b924ff08
use libtaos::*;
use taos::sync::*;
fn taos_connect() -> Result<Taos, Error> {
TaosCfgBuilder::default()
.ip("localhost")
.user("root")
.pass("taosdata")
.db("power")
.port(6030u16)
.build()
.expect("TaosCfg builder error")
.connect()
}
#[tokio::main]
async fn main() -> Result<(), Error> {
let taos = taos_connect().expect("connect error");
let result = taos.query("SELECT ts, current FROM meters LIMIT 2").await?;
fn main() -> anyhow::Result<()> {
let taos = TaosBuilder::from_dsn("ws:///power")?.build()?;
let mut result = taos.query("SELECT ts, current FROM meters LIMIT 2")?;
// print column names
let meta: Vec<ColumnMeta> = result.column_meta;
for column in meta {
print!("{}\t", column.name)
}
println!();
let meta = result.fields();
println!("{}", meta.iter().map(|field| field.name()).join("\t"));
// print rows
let rows: Vec<Vec<Field>> = result.rows;
let rows = result.rows();
for row in rows {
for field in row {
print!("{}\t", field);
let row = row?;
for (_name, value) in row {
print!("{}\t", value);
}
println!();
}
Ok(())
}
// output:
// output(suppose you are in +8 timezone):
// ts current
// 2022-03-28 09:56:51.249 10.3
// 2022-03-28 09:56:51.749 12.6
// 2018-10-03T14:38:05+08:00 10.3
// 2018-10-03T14:38:15+08:00 12.6
docs/examples/rust/schemalessexample/Cargo.toml 已删除 100644 → 0
浏览文件 @ 5edce328
[package]
name = "schemalessexample"
version = "0.1.0"
edition = "2021"
[dependencies]
libtaos = { version = "0.4.3" }
docs/examples/rust/schemalessexample/examples/influxdb_line_example.rs 已删除 100644 → 0
浏览文件 @ 5edce328
use libtaos::schemaless::*;
use libtaos::*;
fn main() {
let taos = TaosCfg::default().connect().expect("fail to connect");
taos.raw_query("CREATE DATABASE test").unwrap();
taos.raw_query("USE test").unwrap();
let lines = ["meters,location=California.LosAngeles,groupid=2 current=11.8,voltage=221,phase=0.28 1648432611249",
"meters,location=California.LosAngeles,groupid=2 current=13.4,voltage=223,phase=0.29 1648432611250",
"meters,location=California.LosAngeles,groupid=3 current=10.8,voltage=223,phase=0.29 1648432611249",
"meters,location=California.LosAngeles,groupid=3 current=11.3,voltage=221,phase=0.35 1648432611250"];
let affected_rows = taos
.schemaless_insert(
&lines,
TSDB_SML_LINE_PROTOCOL,
TSDB_SML_TIMESTAMP_MILLISECONDS,
)
.unwrap();
println!("affected_rows={}", affected_rows);
}
// run with: cargo run --example influxdb_line_example
docs/examples/rust/schemalessexample/examples/opentsdb_json_example.rs 已删除 100644 → 0
浏览文件 @ 5edce328
use libtaos::schemaless::*;
use libtaos::*;
fn main() {
let taos = TaosCfg::default().connect().expect("fail to connect");
taos.raw_query("CREATE DATABASE test").unwrap();
taos.raw_query("USE test").unwrap();
let lines = [
r#"[{"metric": "meters.current", "timestamp": 1648432611249, "value": 10.3, "tags": {"location": "California.SanFrancisco", "groupid": 2}},
{"metric": "meters.voltage", "timestamp": 1648432611249, "value": 219, "tags": {"location": "California.LosAngeles", "groupid": 1}},
{"metric": "meters.current", "timestamp": 1648432611250, "value": 12.6, "tags": {"location": "California.SanFrancisco", "groupid": 2}},
{"metric": "meters.voltage", "timestamp": 1648432611250, "value": 221, "tags": {"location": "California.LosAngeles", "groupid": 1}}]"#,
];
let affected_rows = taos
.schemaless_insert(
&lines,
TSDB_SML_JSON_PROTOCOL,
TSDB_SML_TIMESTAMP_NOT_CONFIGURED,
)
.unwrap();
println!("affected_rows={}", affected_rows); // affected_rows=4
}
// run with: cargo run --example opentsdb_json_example
docs/examples/rust/schemalessexample/examples/opentsdb_telnet_example.rs 已删除 100644 → 0
浏览文件 @ 5edce328
use libtaos::schemaless::*;
use libtaos::*;
fn main() {
let taos = TaosCfg::default().connect().expect("fail to connect");
taos.raw_query("CREATE DATABASE test").unwrap();
taos.raw_query("USE test").unwrap();
let lines = [
"meters.current 1648432611249 10.3 location=California.SanFrancisco groupid=2",
"meters.current 1648432611250 12.6 location=California.SanFrancisco groupid=2",
"meters.current 1648432611249 10.8 location=California.LosAngeles groupid=3",
"meters.current 1648432611250 11.3 location=California.LosAngeles groupid=3",
"meters.voltage 1648432611249 219 location=California.SanFrancisco groupid=2",
"meters.voltage 1648432611250 218 location=California.SanFrancisco groupid=2",
"meters.voltage 1648432611249 221 location=California.LosAngeles groupid=3",
"meters.voltage 1648432611250 217 location=California.LosAngeles groupid=3",
];
let affected_rows = taos
.schemaless_insert(
&lines,
TSDB_SML_TELNET_PROTOCOL,
TSDB_SML_TIMESTAMP_NOT_CONFIGURED,
)
.unwrap();
println!("affected_rows={}", affected_rows); // affected_rows=8
}
// run with: cargo run --example opentsdb_telnet_example
docs/zh/01-index.md
浏览文件 @ b924ff08
......@@ -10,7 +10,7 @@ TDengine 充分利用了时序数据的特点，提出了“一个数据采集
如果你是开发者，请一定仔细阅读[开发指南](./develop)一章，该部分对数据库连接、建模、插入数据、查询、流式计算、缓存、数据订阅、用户自定义函数等功能都做了详细介绍，并配有各种编程语言的示例代码。大部分情况下，你只要把示例代码拷贝粘贴，针对自己的应用稍作改动，就能跑起来。
我们已经生活在大数据的时代，纵向扩展已经无法满足日益增长的业务需求，任何系统都必须具有水平扩展的能力，集群成为大数据以及 database 系统的不可缺失功能。TDengine 团队不仅实现了集群功能，而且将这一重要核心功能开源。怎么部署、管理和维护 TDengine 集群，请参考[集群管理](./cluster)一章。
我们已经生活在大数据的时代，纵向扩展已经无法满足日益增长的业务需求，任何系统都必须具有水平扩展的能力，集群成为大数据以及 database 系统的不可缺失功能。TDengine 团队不仅实现了集群功能，而且将这一重要核心功能开源。怎么部署、管理和维护 TDengine 集群，请参考[部署集群](./deployment)一章。
TDengine 采用 SQL 作为其查询语言，大大降低学习成本、降低迁移成本，但同时针对时序数据场景，又做了一些扩展，以支持插值、降采样、时间加权平均等操作。[SQL 手册](./taos-sql)一章详细描述了 SQL 语法、详细列出了各种支持的命令和函数。
......
docs/zh/02-intro.md
浏览文件 @ b924ff08
......@@ -3,7 +3,7 @@ title: 产品简介
toc_max_heading_level: 2
---
TDengine 是一款[开源](https://www.taosdata.com/tdengine/open_source_time-series_database)、[高性能](https://www.taosdata.com/tdengine/fast)、[云原生](https://www.taosdata.com/tdengine/cloud_native_time-series_database)的时序数据库 (Time-Series Database, TSDB)。TDengine 能被广泛运用于物联网、工业互联网、车联网、IT 运维、金融等领域。除核心的时序数据库功能外，TDengine 还提供[缓存](/develop/cache/)、[数据订阅](/develop/subscribe)、[流式计算](/develop/continuous-query)等功能，是一极简的时序数据处理平台，最大程度的减小系统设计的复杂度，降低研发和运营成本。
TDengine 是一款[开源](https://www.taosdata.com/tdengine/open_source_time-series_database)、[高性能](https://www.taosdata.com/tdengine/fast)、[云原生](https://www.taosdata.com/tdengine/cloud_native_time-series_database)的时序数据库 (Time-Series Database, TSDB)。TDengine 能被广泛运用于物联网、工业互联网、车联网、IT 运维、金融等领域。除核心的时序数据库功能外，TDengine 还提供[缓存](../develop/cache/)、[数据订阅](../develop/subscribe)、[流式计算](../develop/continuous-query)等功能，是一极简的时序数据处理平台，最大程度的减小系统设计的复杂度，降低研发和运营成本。
本章节介绍TDengine的主要功能、竞争优势、适用场景、与其他数据库的对比测试等等，让大家对TDengine有个整体的了解。
......@@ -11,21 +11,22 @@ TDengine 是一款[开源](https://www.taosdata.com/tdengine/open_source_time-se
TDengine的主要功能如下：
1. 高速数据写入，除 [SQL 写入](/develop/insert-data/sql-writing)外，还支持 [Schemaless 写入](/reference/schemaless/)，支持 [InfluxDB LINE 协议](/develop/insert-data/influxdb-line)，[OpenTSDB Telnet](/develop/insert-data/opentsdb-telnet), [OpenTSDB JSON ](/develop/insert-data/opentsdb-json)等协议写入；
2. 第三方数据采集工具 [Telegraf](/third-party/telegraf)，[Prometheus](/third-party/prometheus)，[StatsD](/third-party/statsd)，[collectd](/third-party/collectd)，[icinga2](/third-party/icinga2), [TCollector](/third-party/tcollector), [EMQ](/third-party/emq-broker), [HiveMQ](/third-party/hive-mq-broker) 等都可以进行配置后，不用任何代码，即可将数据写入；
3. 支持[各种查询](/develop/query-data),包括聚合查询、嵌套查询、降采样查询、插值等
4. 支持[用户自定义函数](/develop/udf)
5. 支持[缓存](/develop/cache)，将每张表的最后一条记录缓存起来，这样无需 Redis
6. 支持[流式计算](/develop/continuous-query)(Stream Processing)
7. 支持[数据订阅](/develop/subscribe)，而且可以指定过滤条件
8. 支持[集群](/cluster/)，可以通过多节点进行水平扩展，并通过多副本实现高可靠
9. 提供[命令行程序](/reference/taos-shell)，便于管理集群，检查系统状态，做即席查询
10. 提供多种数据的[导入](/operation/import)、[导出](/operation/export)
11. 支持对[TDengine 集群本身的监控](/operation/monitor)
12. 提供 [C/C++](/reference/connector/cpp), [Java](/reference/connector/java), [Python](/reference/connector/python), [Go](/reference/connector/go), [Rust](/reference/connector/rust), [Node.js](/reference/connector/node) 等多种编程语言的[连接器](/reference/connector/)
13. 支持 [REST 接口](/reference/rest-api/)
14. 支持与[ Grafana 无缝集成](/third-party/grafana)
1. 高速数据写入，除 [SQL 写入](../develop/insert-data/sql-writing)外，还支持 [Schemaless 写入](../reference/schemaless/)，支持 [InfluxDB LINE 协议](../develop/insert-data/influxdb-line)，[OpenTSDB Telnet](../develop/insert-data/opentsdb-telnet), [OpenTSDB JSON ](../develop/insert-data/opentsdb-json)等协议写入；
2. 第三方数据采集工具 [Telegraf](../third-party/telegraf)，[Prometheus](../third-party/prometheus)，[StatsD](../third-party/statsd)，[collectd](../third-party/collectd)，[icinga2](../third-party/icinga2), [TCollector](../third-party/tcollector), [EMQ](../third-party/emq-broker), [HiveMQ](../third-party/hive-mq-broker) 等都可以进行配置后，不用任何代码，即可将数据写入；
3. 支持[各种查询](../develop/query-data),包括聚合查询、嵌套查询、降采样查询、插值等
4. 支持[用户自定义函数](../develop/udf)
5. 支持[缓存](../develop/cache)，将每张表的最后一条记录缓存起来，这样无需 Redis
6. 支持[流式计算](../develop/stream)(Stream Processing)
7. 支持[数据订阅](../develop/tmq)，而且可以指定过滤条件
8. 支持[集群](../deployment/)，可以通过多节点进行水平扩展，并通过多副本实现高可靠
9. 提供[命令行程序](../reference/taos-shell)，便于管理集群，检查系统状态，做即席查询
10. 提供多种数据的[导入](../operation/import)、[导出](../operation/export)
11. 支持对[TDengine 集群本身的监控](../operation/monitor)
12. 提供 [C/C++](../reference/connector/cpp), [Java](../reference/connector/java), [Python](../reference/connector/python), [Go](../reference/connector/go), [Rust](../reference/connector/rust), [Node.js](../reference/connector/node) 等多种编程语言的[连接器](../reference/connector/)
13. 支持 [REST 接口](../reference/rest-api/)
14. 支持与[ Grafana 无缝集成](../third-party/grafana)
15. 支持与 Google Data Studio 无缝集成
16. 支持 [Kubenetes 部署](../deployment/k8s)
更多细小的功能，请阅读整个文档。
......
docs/zh/07-develop/03-insert-data/02-influxdb-line.mdx
浏览文件 @ b924ff08
......@@ -54,9 +54,6 @@ meters,location=California.LosAngeles,groupid=2 current=13.4,voltage=223,phase=0
<TabItem label="Go" value="go">
<GoLine />
</TabItem>
<TabItem label="Rust" value="rust">
<RustLine />
</TabItem>
<TabItem label="Node.js" value="nodejs">
<NodeLine />
</TabItem>
......
docs/zh/07-develop/03-insert-data/03-opentsdb-telnet.mdx
浏览文件 @ b924ff08
......@@ -46,9 +46,6 @@ meters.current 1648432611250 11.3 location=California.LosAngeles groupid=3
<TabItem label="Go" value="go">
<GoTelnet />
</TabItem>
<TabItem label="Rust" value="rust">
<RustTelnet />
</TabItem>
<TabItem label="Node.js" value="nodejs">
<NodeTelnet />
</TabItem>
......
docs/zh/07-develop/03-insert-data/04-opentsdb-json.mdx
浏览文件 @ b924ff08
......@@ -63,9 +63,6 @@ OpenTSDB JSON 格式协议采用一个 JSON 字符串表示一行或多行数据
<TabItem label="Go" value="go">
<GoJson />
</TabItem>
<TabItem label="Rust" value="rust">
<RustJson />
</TabItem>
<TabItem label="Node.js" value="nodejs">
<NodeJson />
</TabItem>
......
docs/zh/07-develop/03-insert-data/_rust_line.mdx
浏览文件 @ b924ff08
```rust
{{#include docs/examples/rust/schemalessexample/examples/influxdb_line_example.rs}}
```
docs/zh/07-develop/03-insert-data/_rust_opts_json.mdx
浏览文件 @ b924ff08
```rust
{{#include docs/examples/rust/schemalessexample/examples/opentsdb_json_example.rs}}
```
docs/zh/07-develop/03-insert-data/_rust_opts_telnet.mdx
浏览文件 @ b924ff08
```rust
{{#include docs/examples/rust/schemalessexample/examples/opentsdb_telnet_example.rs}}
```
docs/zh/10-deployment/03-k8s.md
浏览文件 @ b924ff08
......@@ -3,11 +3,20 @@ sidebar_label: Kubernetes
title: 在 Kubernetes 上部署 TDengine 集群
---
以下配置文件可以从 [GitHub 仓库](https://github.com/taosdata/TDengine-Operator/tree/3.0/src/tdengine) 下载。
作为面向云原生架构设计的时序数据库，TDengine 支持 Kubernetes 部署。这里介绍如何使用 YAML 文件一步一步从头创建一个 TDengine 集群，并重点介绍 Kubernetes 环境下 TDengine 的常用操作。
## 前置条件
要使用 Kubernetes 部署管理 TDengine 集群，需要做好如下准备工作。
* 本文和下一章使用 minikube、kubectl 和 helm 等工具进行安装部署，请提前安装好相应软件
* Kubernetes 已经安装部署并能正常访问使用或更新必要的容器仓库或其他服务
以下配置文件也可以从 [GitHub 仓库](https://github.com/taosdata/TDengine-Operator/tree/3.0/src/tdengine) 下载。
## 配置 Service 服务
创建一个 Service 配置文件：`taosd-service.yaml`，服务名称 `metadata.name` (此处为 "taosd") 将在下一步中使用到。添加 TDengine 所用到的所有端口：
创建一个 Service 配置文件：`taosd-service.yaml`，服务名称 `metadata.name` (此处为 "taosd") 将在下一步中使用到。添加 TDengine 所用到的端口：
```yaml
---
......@@ -31,7 +40,8 @@ spec:
## 有状态服务 StatefulSet
根据 Kubernetes 对各类部署的说明，我们将使用 StatefulSet 作为 TDengine 的服务类型，创建文件 `tdengine.yaml`：
根据 Kubernetes 对各类部署的说明，我们将使用 StatefulSet 作为 TDengine 的服务类型。
创建文件 `tdengine.yaml`，其中 replicas 定义集群节点的数量为 3。节点时区为中国（Asia/Shanghai），每个节点分配 10G 标准（standard）存储。你也可以根据实际情况进行相应修改。
```yaml
---
......@@ -43,7 +53,7 @@ metadata:
app: "tdengine"
spec:
serviceName: "taosd"
replicas: 2
replicas: 3
updateStrategy:
type: RollingUpdate
selector:
......@@ -58,10 +68,7 @@ spec:
containers:
- name: "tdengine"
image: "tdengine/tdengine:3.0.0.0"
imagePullPolicy: "Always"
envFrom:
- configMapRef:
name: taoscfg
imagePullPolicy: "IfNotPresent"
ports:
- name: tcp6030
protocol: "TCP"
......@@ -130,10 +137,9 @@ spec:
```bash
kubectl apply -f taosd-service.yaml
kubectl apply -f tdengine.yaml
```
上面的配置将生成一个三节点的 TDengine 集群，dnode 是自动配置的，可以使用 show dnodes 命令查看当前集群的节点：
上面的配置将生成一个三节点的 TDengine 集群，dnode 为自动配置，可以使用 show dnodes 命令查看当前集群的节点：
```bash
kubectl exec -i -t tdengine-0 -- taos -s "show dnodes"
......
docs/zh/14-reference/13-schemaless/13-schemaless.md
浏览文件 @ b924ff08
......@@ -3,8 +3,7 @@ title: Schemaless 写入
description: 'Schemaless 写入方式，可以免于预先创建超级表/子表的步骤，随着数据写入接口能够自动创建与数据对应的存储结构'
---
在物联网应用中，常会采集比较多的数据项，用于实现智能控制、业务分析、设备监控等。由于应用逻辑的版本升级，或者设备自身的硬件调整等原因，数据采集项就有可能比较频繁地出现变动。为了在这种情况下方便地完成数据记录工作，TDengine
从 2.2.0.0 版本开始，提供调用 Schemaless 写入方式，可以免于预先创建超级表/子表的步骤，随着数据写入接口能够自动创建与数据对应的存储结构。并且在必要时，Schemaless
在物联网应用中，常会采集比较多的数据项，用于实现智能控制、业务分析、设备监控等。由于应用逻辑的版本升级，或者设备自身的硬件调整等原因，数据采集项就有可能比较频繁地出现变动。为了在这种情况下方便地完成数据记录工作，TDengine提供调用 Schemaless 写入方式，可以免于预先创建超级表/子表的步骤，随着数据写入接口能够自动创建与数据对应的存储结构。并且在必要时，Schemaless
将自动增加必要的数据列，保证用户写入的数据可以被正确存储。
无模式写入方式建立的超级表及其对应的子表与通过 SQL 直接建立的超级表和子表完全没有区别，你也可以通过，SQL 语句直接向其中写入数据。需要注意的是，通过无模式写入方式建立的表，其表名是基于标签值按照固定的映射规则生成，所以无法明确地进行表意，缺乏可读性。
......@@ -41,10 +40,10 @@ tag_set 中的所有的数据自动转化为 nchar 数据类型，并不需要
| -------- | -------- | ------------ | -------------- |
| 1 | 无或 f64 | double | 8 |
| 2 | f32 | float | 4 |
| 3 | i8 | TinyInt | 1 |
| 4 | i16 | SmallInt | 2 |
| 5 | i32 | Int | 4 |
| 6 | i64 或 i | Bigint | 8 |
| 3 | i8/u8 | TinyInt/UTinyInt | 1 |
| 4 | i16/u16 | SmallInt/USmallInt | 2 |
| 5 | i32/u32 | Int/UInt | 4 |
| 6 | i64/i/u64/u | BigInt/BigInt/UBigInt/UBigInt | 8 |
- t, T, true, True, TRUE, f, F, false, False 将直接作为 BOOL 型来处理。
......@@ -69,20 +68,21 @@ st,t1=3,t2=4,t3=t3 c1=3i64,c3="passit",c2=false,c4=4f64 1626006833639000000
```
需要注意的是，这里的 tag_key1, tag_key2 并不是用户输入的标签的原始顺序，而是使用了标签名称按照字符串升序排列后的结果。所以，tag_key1 并不是在行协议中输入的第一个标签。
排列完成以后计算该字符串的 MD5 散列值 "md5_val"。然后将计算的结果与字符串组合生成表名：“t_md5_val”。其中的 “t\*” 是固定的前缀，每个通过该映射关系自动生成的表都具有该前缀。
排列完成以后计算该字符串的 MD5 散列值 "md5_val"。然后将计算的结果与字符串组合生成表名：“t_md5_val”。其中的 “t_” 是固定的前缀，每个通过该映射关系自动生成的表都具有该前缀。
为了让用户可以指定生成的表名，可以通过配置smlChildTableName来指定（比如 配置smlChildTableName=tname 插入数据为st,tname=cpu1,t1=4 c1=3 1626006833639000000 则创建的表名为cpu1，注意如果多行数据tname相同，但是后面的tag_set不同，则使用第一次自动建表时指定的tag_set，其他的会忽略）。
2. 如果解析行协议获得的超级表不存在，则会创建这个超级表。
2. 如果解析行协议获得的超级表不存在，则会创建这个超级表（不建议手动创建超级表，不然插入数据可能异常）。
3. 如果解析行协议获得子表不存在，则 Schemaless 会按照步骤 1 或 2 确定的子表名来创建子表。
4. 如果数据行中指定的标签列或普通列不存在，则在超级表中增加对应的标签列或普通列（只增不减）。
5. 如果超级表中存在一些标签列或普通列未在一个数据行中被指定取值，那么这些列的值在这一行中会被置为
NULL。
6. 对 BINARY 或 NCHAR 列，如果数据行中所提供值的长度超出了列类型的限制，自动增加该列允许存储的字符长度上限（只增不减），以保证数据的完整保存。
7. 如果指定的数据子表已经存在，而且本次指定的标签列取值跟已保存的值不一样，那么最新的数据行中的值会覆盖旧的标签列取值。
8. 整个处理过程中遇到的错误会中断写入过程，并返回错误代码。
7. 整个处理过程中遇到的错误会中断写入过程，并返回错误代码。
8. 为了提高写入的效率，默认假设同一个超级表中field_set的顺序是一样的（第一条数据包含所有的field，后面的数据按照这个顺序），如果顺序不一样，需要配置参数smlDataFormat为false，否则，数据写入按照相同顺序写入，库中数据会异常。
:::tip
无模式所有的处理逻辑，仍会遵循 TDengine 对数据结构的底层限制，例如每行数据的总长度不能超过
48KB。这方面的具体限制约束请参见 [TAOS SQL 边界限制](/taos-sql/limit)
16KB。这方面的具体限制约束请参见 [TAOS SQL 边界限制](/taos-sql/limit)
:::
......
include/common/tdatablock.h
浏览文件 @ b924ff08
......@@ -249,6 +249,7 @@ char* dumpBlockData(SSDataBlock* pDataBlock, const char* flag, char** dumpBuf);
int32_t buildSubmitReqFromDataBlock(SSubmitReq** pReq, const SSDataBlock* pDataBlocks, STSchema* pTSchema, int32_t vgId,
tb_uid_t suid);
char* buildCtbNameByGroupId(const char* stbName, uint64_t groupId);
static FORCE_INLINE int32_t blockGetEncodeSize(const SSDataBlock* pBlock) {
......
include/libs/nodes/querynodes.h
浏览文件 @ b924ff08
......@@ -269,6 +269,7 @@ typedef struct SSelectStmt {
bool hasInterpFunc;
bool hasLastRowFunc;
bool hasTimeLineFunc;
bool hasUdaf;
bool onlyHasKeepOrderFunc;
bool groupSort;
} SSelectStmt;
......
packaging/deb/DEBIAN/control
浏览文件 @ b924ff08
Package: tdengine
Version: 1.0.0
Version: 3.0.0
Section: utils
Priority: optional
#Essential: no
......
packaging/deb/DEBIAN/preinst
浏览文件 @ b924ff08
#!/bin/bash
if [ -f /var/lib/taos/dnode/dnodeCfg.json ]; then
echo -e "The default data directory \033[41;37m/var/lib/taos\033[0m contains old data of tdengine 2.x, please clear it before installing!"
exit 1
fi
csudo=""
if command -v sudo > /dev/null; then
csudo="sudo "
......
packaging/deb/DEBIAN/prerm
浏览文件 @ b924ff08
#!/bin/bash
if [ $1 -eq "abort-upgrade" ]; then
exit 0
fi
insmetaPath="/usr/local/taos/script"
csudo=""
......
packaging/rpm/tdengine.spec
浏览文件 @ b924ff08
......@@ -132,6 +132,10 @@ fi
#Scripts executed before installation
%pre
if [ -f /var/lib/taos/dnode/dnodeCfg.json ]; then
echo -e "The default data directory \033[41;37m/var/lib/taos\033[0m contains old data of tdengine 2.x, please clear it before installing!"
exit 1
fi
csudo=""
if command -v sudo > /dev/null; then
csudo="sudo "
......
source/common/CMakeLists.txt
浏览文件 @ b924ff08
......@@ -5,6 +5,10 @@ if (DEFINED GRANT_CFG_INCLUDE_DIR)
add_definitions(-DGRANTS_CFG)
endif()
IF (TD_GRANT)
ADD_DEFINITIONS(-D_GRANT)
ENDIF ()
target_include_directories(
common
PUBLIC "${TD_SOURCE_DIR}/include/common"
......
source/common/src/systable.c
浏览文件 @ b924ff08
......@@ -346,7 +346,7 @@ static const SSysTableMeta perfsMeta[] = {
{TSDB_PERFS_TABLE_TOPICS, topicSchema, tListLen(topicSchema)},
{TSDB_PERFS_TABLE_CONSUMERS, consumerSchema, tListLen(consumerSchema)},
{TSDB_PERFS_TABLE_SUBSCRIPTIONS, subscriptionSchema, tListLen(subscriptionSchema)},
{TSDB_PERFS_TABLE_OFFSETS, offsetSchema, tListLen(offsetSchema)},
// {TSDB_PERFS_TABLE_OFFSETS, offsetSchema, tListLen(offsetSchema)},
{TSDB_PERFS_TABLE_TRANS, transSchema, tListLen(transSchema)},
{TSDB_PERFS_TABLE_SMAS, smaSchema, tListLen(smaSchema)},
{TSDB_PERFS_TABLE_STREAMS, streamSchema, tListLen(streamSchema)},
......
source/common/src/tdatablock.c
浏览文件 @ b924ff08
......@@ -1713,7 +1713,7 @@ void blockDebugShowDataBlocks(const SArray* dataBlocks, const char* flag) {
char pBuf[128] = {0};
int32_t sz = taosArrayGetSize(dataBlocks);
for (int32_t i = 0; i < sz; i++) {
SSDataBlock* pDataBlock = taosArrayGet(dataBlocks, i);
SSDataBlock* pDataBlock = taosArrayGetP(dataBlocks, i);
size_t numOfCols = taosArrayGetSize(pDataBlock->pDataBlock);
int32_t rows = pDataBlock->info.rows;
......@@ -1870,10 +1870,10 @@ char* dumpBlockData(SSDataBlock* pDataBlock, const char* flag, char** pDataBuf)
* @brief TODO: Assume that the final generated result it less than 3M
*
* @param pReq
* @param pDataBlock
* @param pDataBlocks
* @param vgId
* @param suid
*
*
*/
int32_t buildSubmitReqFromDataBlock(SSubmitReq** pReq, const SSDataBlock* pDataBlock, STSchema* pTSchema, int32_t vgId,
tb_uid_t suid) {
......
source/dnode/mgmt/mgmt_vnode/src/vmHandle.c
浏览文件 @ b924ff08
......@@ -337,6 +337,7 @@ SArray *vmGetMsgHandles() {
if (dmSetMgmtHandle(pArray, TDMT_SCH_QUERY, vmPutMsgToQueryQueue, 0) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_SCH_MERGE_QUERY, vmPutMsgToQueryQueue, 0) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_SCH_QUERY_CONTINUE, vmPutMsgToQueryQueue, 0) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_VND_FETCH_RSMA, vmPutMsgToQueryQueue, 0) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_SCH_FETCH, vmPutMsgToFetchQueue, 0) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_SCH_MERGE_FETCH, vmPutMsgToFetchQueue, 0) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_VND_ALTER_TABLE, vmPutMsgToWriteQueue, 0) == NULL) goto _OVER;
......@@ -347,7 +348,6 @@ SArray *vmGetMsgHandles() {
if (dmSetMgmtHandle(pArray, TDMT_VND_TABLES_META, vmPutMsgToFetchQueue, 0) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_SCH_CANCEL_TASK, vmPutMsgToFetchQueue, 0) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_SCH_DROP_TASK, vmPutMsgToFetchQueue, 0) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_VND_FETCH_RSMA, vmPutMsgToFetchQueue, 0) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_VND_CREATE_STB, vmPutMsgToWriteQueue, 0) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_VND_DROP_TTL_TABLE, vmPutMsgToWriteQueue, 0) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_VND_ALTER_STB, vmPutMsgToWriteQueue, 0) == NULL) goto _OVER;
......
source/dnode/mgmt/mgmt_vnode/src/vmWorker.c
浏览文件 @ b924ff08
......@@ -54,7 +54,7 @@ static void vmProcessMgmtQueue(SQueueInfo *pInfo, SRpcMsg *pMsg) {
if (IsReq(pMsg)) {
if (code != 0) {
if (terrno != 0) code = terrno;
dGError("msg:%p, failed to process since %s", pMsg, terrstr());
dGError("msg:%p, failed to process since %s", pMsg, terrstr(code));
}
vmSendRsp(pMsg, code);
}
......@@ -72,7 +72,7 @@ static void vmProcessQueryQueue(SQueueInfo *pInfo, SRpcMsg *pMsg) {
int32_t code = vnodeProcessQueryMsg(pVnode->pImpl, pMsg);
if (code != 0) {
if (terrno != 0) code = terrno;
dGError("vgId:%d, msg:%p failed to query since %s", pVnode->vgId, pMsg, terrstr());
dGError("vgId:%d, msg:%p failed to query since %s", pVnode->vgId, pMsg, terrstr(code));
vmSendRsp(pMsg, code);
}
......@@ -89,7 +89,7 @@ static void vmProcessStreamQueue(SQueueInfo *pInfo, SRpcMsg *pMsg) {
int32_t code = vnodeProcessFetchMsg(pVnode->pImpl, pMsg, pInfo);
if (code != 0) {
if (terrno != 0) code = terrno;
dGError("vgId:%d, msg:%p failed to process stream since %s", pVnode->vgId, pMsg, terrstr());
dGError("vgId:%d, msg:%p failed to process stream since %s", pVnode->vgId, pMsg, terrstr(code));
vmSendRsp(pMsg, code);
}
......@@ -110,7 +110,7 @@ static void vmProcessFetchQueue(SQueueInfo *pInfo, STaosQall *qall, int32_t numO
int32_t code = vnodeProcessFetchMsg(pVnode->pImpl, pMsg, pInfo);
if (code != 0) {
if (terrno != 0) code = terrno;
dGError("vgId:%d, msg:%p failed to fetch since %s", pVnode->vgId, pMsg, terrstr());
dGError("vgId:%d, msg:%p failed to fetch since %s", pVnode->vgId, pMsg, terrstr(code));
vmSendRsp(pMsg, code);
}
......@@ -156,7 +156,7 @@ static int32_t vmPutMsgToQueue(SVnodeMgmt *pMgmt, SRpcMsg *pMsg, EQueueType qtyp
if ((pMsg->msgType == TDMT_SCH_QUERY) && (grantCheck(TSDB_GRANT_TIME) != TSDB_CODE_SUCCESS)) {
terrno = TSDB_CODE_GRANT_EXPIRED;
code = terrno;
dDebug("vgId:%d, msg:%p put into vnode-query queue failed since %s", pVnode->vgId, pMsg, terrstr());
dDebug("vgId:%d, msg:%p put into vnode-query queue failed since %s", pVnode->vgId, pMsg, terrstr(code));
} else {
vnodePreprocessQueryMsg(pVnode->pImpl, pMsg);
dGTrace("vgId:%d, msg:%p put into vnode-query queue", pVnode->vgId, pMsg);
......@@ -179,11 +179,11 @@ static int32_t vmPutMsgToQueue(SVnodeMgmt *pMgmt, SRpcMsg *pMsg, EQueueType qtyp
if (!osDataSpaceAvailable()) {
terrno = TSDB_CODE_VND_NO_DISKSPACE;
code = terrno;
dError("vgId:%d, msg:%p put into vnode-write queue failed since %s", pVnode->vgId, pMsg, terrstr());
dError("vgId:%d, msg:%p put into vnode-write queue failed since %s", pVnode->vgId, pMsg, terrstr(code));
} else if ((pMsg->msgType == TDMT_VND_SUBMIT) && (grantCheck(TSDB_GRANT_STORAGE) != TSDB_CODE_SUCCESS)) {
terrno = TSDB_CODE_VND_NO_WRITE_AUTH;
code = terrno;
dDebug("vgId:%d, msg:%p put into vnode-write queue failed since %s", pVnode->vgId, pMsg, terrstr());
dDebug("vgId:%d, msg:%p put into vnode-write queue failed since %s", pVnode->vgId, pMsg, terrstr(code));
} else {
dGTrace("vgId:%d, msg:%p put into vnode-write queue", pVnode->vgId, pMsg);
taosWriteQitem(pVnode->pWriteQ, pMsg);
......
source/dnode/mnode/impl/CMakeLists.txt
浏览文件 @ b924ff08
......@@ -15,6 +15,7 @@ target_include_directories(
target_link_libraries(
mnode scheduler sdb wal transport cjson sync monitor executor qworker stream parser
)
IF (TD_GRANT)
TARGET_LINK_LIBRARIES(mnode grant)
ADD_DEFINITIONS(-D_GRANT)
......
source/dnode/mnode/impl/src/mndDb.c
浏览文件 @ b924ff08
......@@ -1419,7 +1419,7 @@ const char *mndGetDbStr(const char *src) {
return pos;
}
int64_t getValOfDiffPrecision(int8_t unit, int64_t val) {
static int64_t getValOfDiffPrecision(int8_t unit, int64_t val) {
int64_t v = 0;
switch (unit) {
case 's':
......@@ -1444,7 +1444,7 @@ int64_t getValOfDiffPrecision(int8_t unit, int64_t val) {
return v;
}
char *buildRetension(SArray *pRetension) {
static char *buildRetension(SArray *pRetension) {
size_t size = taosArrayGetSize(pRetension);
if (size == 0) {
return NULL;
......
source/dnode/vnode/CMakeLists.txt
浏览文件 @ b924ff08
......@@ -86,6 +86,11 @@ target_link_libraries(
PUBLIC stream
PUBLIC index
)
IF (TD_GRANT)
TARGET_LINK_LIBRARIES(vnode PUBLIC grant)
ENDIF ()
target_compile_definitions(vnode PUBLIC -DMETA_REFACT)
if(${BUILD_WITH_INVERTEDINDEX})
......
source/dnode/vnode/src/meta/metaTable.c
浏览文件 @ b924ff08
......@@ -209,8 +209,8 @@ int metaCreateSTable(SMeta *pMeta, int64_t version, SVCreateStbReq *pReq) {
return 0;
_err:
metaError("vgId:%d, failed to create stb:%s uid:%" PRId64 " since %s", TD_VID(pMeta->pVnode), pReq->name,
pReq->suid, tstrerror(terrno));
metaError("vgId:%d, failed to create stb:%s uid:%" PRId64 " since %s", TD_VID(pMeta->pVnode), pReq->name, pReq->suid,
tstrerror(terrno));
return -1;
}
......@@ -304,7 +304,8 @@ int metaAlterSTable(SMeta *pMeta, int64_t version, SVCreateStbReq *pReq) {
ret = tdbTbcGet(pUidIdxc, NULL, NULL, &pData, &nData);
if (ret < 0) {
ASSERT(0);
terrno = TSDB_CODE_TDB_STB_NOT_EXIST;
// ASSERT(0);
return -1;
}
......@@ -1196,6 +1197,9 @@ static int metaSaveToSkmDb(SMeta *pMeta, const SMetaEntry *pME) {
goto _exit;
}
metaDebug("vgId:%d, set schema:(%" PRId64 ") sver:%d since %s", TD_VID(pMeta->pVnode), pME->uid, pSW->version,
tstrerror(terrno));
_exit:
taosMemoryFree(pVal);
tEncoderClear(&coder);
......
source/dnode/vnode/src/sma/smaRollup.c
浏览文件 @ b924ff08
......@@ -293,7 +293,9 @@ static int32_t tdSetRSmaInfoItemParams(SSma *pSma, SRSmaParam *param, SRSmaStat
if (pItem->maxDelay > TSDB_MAX_ROLLUP_MAX_DELAY) {
pItem->maxDelay = TSDB_MAX_ROLLUP_MAX_DELAY;
}
pItem->level = idx == 0 ? TSDB_RETENTION_L1 : TSDB_RETENTION_L2;
taosTmrReset(tdRSmaFetchTrigger, pItem->maxDelay, pItem, smaMgmt.tmrHandle, &pItem->tmrId);
smaInfo("vgId:%d, table:%" PRIi64 " level:%" PRIi8 " maxdelay:%" PRIi64 " watermark:%" PRIi64
", finally maxdelay:%" PRIi32,
TD_VID(pVnode), pRSmaInfo->suid, idx + 1, param->maxdelay[idx], param->watermark[idx], pItem->maxDelay);
......@@ -613,34 +615,38 @@ static int32_t tdRSmaFetchAndSubmitResult(SSma *pSma, qTaskInfo_t taskInfo, SRSm
while (1) {
uint64_t ts;
int32_t code = qExecTaskOpt(taskInfo, pResList, &ts);
if (code < 0) {
smaError("vgId:%d, qExecTask for rsma table %" PRIi64 " level %" PRIi8 " failed since %s", SMA_VID(pSma), suid,
pItem->level, terrstr(code));
goto _err;
if (code < 0) {
if (code == TSDB_CODE_QRY_IN_EXEC) {
break;
} else {
smaError("vgId:%d, qExecTask for rsma table %" PRIi64 " level %" PRIi8 " failed since %s", SMA_VID(pSma), suid,
pItem->level, terrstr(code));
goto _err;
}
}
if (taosArrayGetSize(pResList) == 0) {
if (terrno == 0) {
smaDebug("vgId:%d, no rsma %" PRIi8 " data fetched yet", SMA_VID(pSma), pItem->level);
// smaDebug("vgId:%d, no rsma %" PRIi8 " data fetched yet", SMA_VID(pSma), pItem->level);
} else {
smaDebug("vgId:%d, no rsma %" PRIi8 " data fetched since %s", SMA_VID(pSma), pItem->level, terrstr());
goto _err;
}
break;
} else {
smaDebug("vgId:%d, rsma %" PRIi8 " data fetched", SMA_VID(pSma), pItem->level);
}
for (int32_t i = 0; i < taosArrayGetSize(pResList); ++i) {
SSDataBlock *output = taosArrayGetP(pResList, i);
#if 1
char flag[10] = {0};
snprintf(flag, 10, "level %" PRIi8, pItem->level);
// blockDebugShowDataBlocks(output, flag);
// taosArrayDestroy(pResult);
char flag[10] = {0};
snprintf(flag, 10, "level %" PRIi8, pItem->level);
blockDebugShowDataBlocks(pResList, flag);
#endif
STsdb * sinkTsdb = (pItem->level == TSDB_RETENTION_L1 ? pSma->pRSmaTsdb[0] : pSma->pRSmaTsdb[1]);
SSubmitReq *pReq = NULL;
for (int32_t i = 0; i < taosArrayGetSize(pResList); ++i) {
SSDataBlock *output = taosArrayGetP(pResList, i);
STsdb *sinkTsdb = (pItem->level == TSDB_RETENTION_L1 ? pSma->pRSmaTsdb[0] : pSma->pRSmaTsdb[1]);
SSubmitReq *pReq = NULL;
// TODO: the schema update should be handled later(TD-17965)
if (buildSubmitReqFromDataBlock(&pReq, output, pTSchema, SMA_VID(pSma), suid) < 0) {
......@@ -655,11 +661,11 @@ static int32_t tdRSmaFetchAndSubmitResult(SSma *pSma, qTaskInfo_t taskInfo, SRSm
SMA_VID(pSma), suid, pItem->level, terrstr());
goto _err;
}
taosMemoryFreeClear(pReq);
smaDebug("vgId:%d, process submit req for rsma table %" PRIi64 " level %" PRIi8 " version:%" PRIi64,
SMA_VID(pSma), suid, pItem->level, output->info.version);
taosMemoryFreeClear(pReq);
}
}
......@@ -692,15 +698,12 @@ static int32_t tdExecuteRSmaImpl(SSma *pSma, const void *pMsg, int32_t inputType
}
SRSmaInfoItem *pItem = RSMA_INFO_ITEM(pInfo, idx);
tdRSmaFetchAndSubmitResult(pSma, RSMA_INFO_QTASK(pInfo, idx), pItem, pInfo->pTSchema, suid,
STREAM_INPUT__DATA_SUBMIT);
atomic_store_8(&pItem->triggerStat, TASK_TRIGGER_STAT_ACTIVE);
if (smaMgmt.tmrHandle) {
taosTmrReset(tdRSmaFetchTrigger, pItem->maxDelay, pItem, smaMgmt.tmrHandle, &pItem->tmrId);
} else {
ASSERT(0);
}
return TSDB_CODE_SUCCESS;
......@@ -746,7 +749,6 @@ static SRSmaInfo *tdAcquireRSmaInfoBySuid(SSma *pSma, int64_t suid) {
return NULL;
}
// clone the SRSmaInfo from iRsmaInfoHash to rsmaInfoHash if in committing stat
SRSmaInfo *pCowRSmaInfo = NULL;
// lock
......@@ -793,13 +795,7 @@ static FORCE_INLINE void tdReleaseRSmaInfo(SSma *pSma, SRSmaInfo *pInfo) {
static int32_t tdExecuteRSma(SSma *pSma, const void *pMsg, int32_t inputType, tb_uid_t suid) {
SRSmaInfo *pRSmaInfo = tdAcquireRSmaInfoBySuid(pSma, suid);
if (!pRSmaInfo) {
smaDebug("vgId:%d, execute rsma, no rsma info for suid:%" PRIu64, SMA_VID(pSma), suid);
return TSDB_CODE_SUCCESS;
}
if (!RSMA_INFO_QTASK(pRSmaInfo, 0)) {
tdReleaseRSmaInfo(pSma, pRSmaInfo);
smaDebug("vgId:%d, execute rsma, no rsma qTaskInfo for suid:%" PRIu64, SMA_VID(pSma), suid);
smaError("vgId:%d, execute rsma, no rsma info for suid:%" PRIu64, SMA_VID(pSma), suid);
return TSDB_CODE_SUCCESS;
}
......@@ -1331,14 +1327,16 @@ static void tdRSmaFetchTrigger(void *param, void *tmrId) {
SRSmaInfo *pRSmaInfo = tdGetRSmaInfoByItem(pItem);
if (RSMA_INFO_IS_DEL(pRSmaInfo)) {
smaDebug("rsma fetch task not start since rsma info already deleted, rsetId:%" PRIi64 " refId:%d)", smaMgmt.rsetId,
pRSmaInfo->refId);
return;
}
SRSmaStat *pStat = (SRSmaStat *)tdAcquireSmaRef(smaMgmt.rsetId, pRSmaInfo->refId);
if (!pStat) {
smaDebug("rsma fetch task not start since already destroyed, rsetId rsetId:%" PRIi64 " refId:%d)", smaMgmt.rsetId,
pRSmaInfo->refId);
smaDebug("rsma fetch task not start since rsma stat already destroyed, rsetId:%" PRIi64 " refId:%d)",
smaMgmt.rsetId, pRSmaInfo->refId);
return;
}
......@@ -1350,8 +1348,8 @@ static void tdRSmaFetchTrigger(void *param, void *tmrId) {
case TASK_TRIGGER_STAT_PAUSED:
case TASK_TRIGGER_STAT_CANCELLED: {
tdReleaseSmaRef(smaMgmt.rsetId, pRSmaInfo->refId);
smaDebug("vgId:%d, not fetch rsma level %" PRIi8 " data since stat is %" PRIi8 ", rsetId rsetId:%" PRIi64
" refId:%d",
smaDebug("vgId:%d, rsma fetch task not start for level %" PRIi8 " since stat is %" PRIi8
", rsetId rsetId:%" PRIi64 " refId:%d",
SMA_VID(pSma), pItem->level, rsmaTriggerStat, smaMgmt.rsetId, pRSmaInfo->refId);
if (rsmaTriggerStat == TASK_TRIGGER_STAT_PAUSED) {
taosTmrReset(tdRSmaFetchTrigger, 5000, pItem, smaMgmt.tmrHandle, &pItem->tmrId);
......@@ -1366,30 +1364,31 @@ static void tdRSmaFetchTrigger(void *param, void *tmrId) {
atomic_val_compare_exchange_8(&pItem->triggerStat, TASK_TRIGGER_STAT_ACTIVE, TASK_TRIGGER_STAT_INACTIVE);
switch (fetchTriggerStat) {
case TASK_TRIGGER_STAT_ACTIVE: {
smaDebug("vgId:%d, fetch rsma level %" PRIi8 " data for table:%" PRIi64 " since stat is active", SMA_VID(pSma),
pItem->level, pRSmaInfo->suid);
smaDebug("vgId:%d, rsma fetch task started for level:%" PRIi8 " suid:%" PRIi64 " since stat is active",
SMA_VID(pSma), pItem->level, pRSmaInfo->suid);
// async process
tdRSmaFetchSend(pSma, pRSmaInfo, pItem->level);
} break;
case TASK_TRIGGER_STAT_PAUSED: {
smaDebug("vgId:%d, not fetch rsma level %" PRIi8 " data for table:%" PRIi64 " since stat is paused",
smaDebug("vgId:%d, rsma fetch task not start for level:%" PRIi8 " suid:%" PRIi64 " since stat is paused",
SMA_VID(pSma), pItem->level, pRSmaInfo->suid);
} break;
case TASK_TRIGGER_STAT_INACTIVE: {
smaDebug("vgId:%d, not fetch rsma level %" PRIi8 " data for table:%" PRIi64 " since stat is inactive",
smaDebug("vgId:%d, rsma fetch task not start for level:%" PRIi8 " suid:%" PRIi64 " since stat is inactive",
SMA_VID(pSma), pItem->level, pRSmaInfo->suid);
} break;
case TASK_TRIGGER_STAT_INIT: {
smaDebug("vgId:%d, not fetch rsma level %" PRIi8 " data for table:%" PRIi64 " since stat is init", SMA_VID(pSma),
pItem->level, pRSmaInfo->suid);
smaDebug("vgId:%d, rsma fetch task not start for level:%" PRIi8 " suid::%" PRIi64 " since stat is init",
SMA_VID(pSma), pItem->level, pRSmaInfo->suid);
} break;
default: {
smaWarn("vgId:%d, not fetch rsma level %" PRIi8 " data for table:%" PRIi64 " since stat is unknown",
smaWarn("vgId:%d, rsma fetch task not start for level:%" PRIi8 " suid:%" PRIi64 " since stat is unknown",
SMA_VID(pSma), pItem->level, pRSmaInfo->suid);
} break;
}
_end:
// taosTmrReset(tdRSmaFetchTrigger, pItem->maxDelay, pItem, smaMgmt.tmrHandle, &pItem->tmrId);
tdReleaseSmaRef(smaMgmt.rsetId, pRSmaInfo->refId);
}
......@@ -1402,7 +1401,7 @@ _end:
* @return int32_t
*/
int32_t tdRSmaFetchSend(SSma *pSma, SRSmaInfo *pInfo, int8_t level) {
SRSmaFetchMsg fetchMsg = { .suid = pInfo->suid, .level = level};
SRSmaFetchMsg fetchMsg = {.suid = pInfo->suid, .level = level};
int32_t ret = 0;
int32_t contLen = 0;
SEncoder encoder = {0};
......@@ -1431,7 +1430,7 @@ int32_t tdRSmaFetchSend(SSma *pSma, SRSmaInfo *pInfo, int8_t level) {
.contLen = contLen,
};
if ((terrno = tmsgPutToQueue(&pSma->pVnode->msgCb, FETCH_QUEUE, &rpcMsg)) != 0) {
if ((terrno = tmsgPutToQueue(&pSma->pVnode->msgCb, QUERY_QUEUE, &rpcMsg)) != 0) {
smaError("vgId:%d, failed to put rsma fetch msg into fetch-queue for suid:%" PRIi64 " level:%" PRIi8 " since %s",
SMA_VID(pSma), pInfo->suid, level, terrstr());
goto _err;
......@@ -1462,7 +1461,7 @@ int32_t smaProcessFetch(SSma *pSma, void *pMsg) {
if (!pRpcMsg || pRpcMsg->contLen < sizeof(SMsgHead)) {
terrno = TSDB_CODE_RSMA_FETCH_MSG_MSSED_UP;
return -1;
goto _err;
}
pBuf = POINTER_SHIFT(pRpcMsg->pCont, sizeof(SMsgHead));
......@@ -1479,7 +1478,7 @@ int32_t smaProcessFetch(SSma *pSma, void *pMsg) {
terrno = TSDB_CODE_RSMA_EMPTY_INFO;
}
smaWarn("vgId:%d, failed to process rsma fetch msg for suid:%" PRIi64 " level:%" PRIi8 " since %s", SMA_VID(pSma),
req.suid, req.level, terrstr());
req.suid, req.level, terrstr());
goto _err;
}
......
source/dnode/vnode/src/tsdb/tsdbRead.c
浏览文件 @ b924ff08
......@@ -13,8 +13,8 @@
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
#include "tsdb.h"
#include "osDef.h"
#include "tsdb.h"
#define ASCENDING_TRAVERSE(o) (o == TSDB_ORDER_ASC)
typedef enum {
......@@ -130,8 +130,8 @@ struct STsdbReader {
SBlockLoadSuppInfo suppInfo;
STsdbReadSnap* pReadSnap;
SIOCostSummary cost;
STSchema* pSchema;// the newest version schema
STSchema* pMemSchema;// the previous schema for in-memory data, to avoid load schema too many times
STSchema* pSchema; // the newest version schema
STSchema* pMemSchema; // the previous schema for in-memory data, to avoid load schema too many times
SDataFReader* pFileReader;
SVersionRange verRange;
......@@ -1213,17 +1213,17 @@ static int32_t buildDataBlockFromBuf(STsdbReader* pReader, STableBlockScanInfo*
return code;
}
static bool tryCopyDistinctRowFromFileBlock(STsdbReader* pReader, SBlockData* pBlockData, int64_t key, SFileBlockDumpInfo* pDumpInfo) {
static bool tryCopyDistinctRowFromFileBlock(STsdbReader* pReader, SBlockData* pBlockData, int64_t key,
SFileBlockDumpInfo* pDumpInfo) {
// opt version
// 1. it is not a border point
// 2. the direct next point is not an duplicated timestamp
if ((pDumpInfo->rowIndex < pDumpInfo->totalRows - 1 && pReader->order == TSDB_ORDER_ASC) ||
(pDumpInfo->rowIndex > 0 && pReader->order == TSDB_ORDER_DESC)) {
int32_t step = pReader->order == TSDB_ORDER_ASC? 1:-1;
int32_t step = pReader->order == TSDB_ORDER_ASC ? 1 : -1;
int64_t nextKey = pBlockData->aTSKEY[pDumpInfo->rowIndex + step];
if (nextKey != key) { // merge is not needed
if (nextKey != key) { // merge is not needed
doAppendRowFromBlock(pReader->pResBlock, pReader, pBlockData, pDumpInfo->rowIndex);
pDumpInfo->rowIndex += step;
return true;
......@@ -1239,7 +1239,7 @@ static FORCE_INLINE STSchema* doGetSchemaForTSRow(int32_t sversion, STsdbReader*
pReader->pSchema = metaGetTbTSchema(pReader->pTsdb->pVnode->pMeta, uid, -1);
}
if (sversion == pReader->pSchema->version) {
if (pReader->pSchema && sversion == pReader->pSchema->version) {
return pReader->pSchema;
}
......@@ -1265,10 +1265,10 @@ static int32_t doMergeBufAndFileRows(STsdbReader* pReader, STableBlockScanInfo*
SBlockData* pBlockData = &pReader->status.fileBlockData;
SFileBlockDumpInfo* pDumpInfo = &pReader->status.fBlockDumpInfo;
TSDBKEY k = TSDBROW_KEY(pRow);
TSDBROW fRow = tsdbRowFromBlockData(pBlockData, pDumpInfo->rowIndex);
SArray* pDelList = pBlockScanInfo->delSkyline;
bool freeTSRow = false;
TSDBKEY k = TSDBROW_KEY(pRow);
TSDBROW fRow = tsdbRowFromBlockData(pBlockData, pDumpInfo->rowIndex);
SArray* pDelList = pBlockScanInfo->delSkyline;
bool freeTSRow = false;
uint64_t uid = pBlockScanInfo->uid;
// ascending order traverse
......@@ -2153,7 +2153,7 @@ TSDBROW* getValidRow(SIterInfo* pIter, const SArray* pDelList, STsdbReader* pRea
}
TSDBROW* pRow = tsdbTbDataIterGet(pIter->iter);
TSDBKEY key = {.ts = pRow->pTSRow->ts, .version = pRow->version};
TSDBKEY key = {.ts = pRow->pTSRow->ts, .version = pRow->version};
if (outOfTimeWindow(key.ts, &pReader->window)) {
pIter->hasVal = false;
return NULL;
......@@ -2186,7 +2186,6 @@ TSDBROW* getValidRow(SIterInfo* pIter, const SArray* pDelList, STsdbReader* pRea
}
}
int32_t doMergeRowsInBuf(SIterInfo* pIter, uint64_t uid, int64_t ts, SArray* pDelList, SRowMerger* pMerger,
STsdbReader* pReader) {
while (1) {
......@@ -2318,9 +2317,8 @@ int32_t doMergeRowsInFileBlocks(SBlockData* pBlockData, STableBlockScanInfo* pSc
void doMergeMultiRows(TSDBROW* pRow, uint64_t uid, SIterInfo* pIter, SArray* pDelList, STSRow** pTSRow,
STsdbReader* pReader, bool* freeTSRow) {
TSDBROW* pNextRow = NULL;
TSDBROW current = *pRow;
TSDBROW current = *pRow;
{ // if the timestamp of the next valid row has a different ts, return current row directly
pIter->hasVal = tsdbTbDataIterNext(pIter->iter);
......@@ -2350,6 +2348,10 @@ void doMergeMultiRows(TSDBROW* pRow, uint64_t uid, SIterInfo* pIter, SArray* pDe
// get the correct schema for data in memory
STSchema* pTSchema = doGetSchemaForTSRow(TSDBROW_SVERSION(&current), pReader, uid);
if (pReader->pSchema == NULL) {
pReader->pSchema = pTSchema;
}
tRowMergerInit2(&merge, pReader->pSchema, &current, pTSchema);
STSchema* pTSchema1 = doGetSchemaForTSRow(TSDBROW_SVERSION(pNextRow), pReader, uid);
......@@ -2390,8 +2392,8 @@ void doMergeMemIMemRows(TSDBROW* pRow, TSDBROW* piRow, STableBlockScanInfo* pBlo
tRowMergerGetRow(&merge, pTSRow);
}
int32_t tsdbGetNextRowInMem(STableBlockScanInfo* pBlockScanInfo, STsdbReader* pReader, STSRow** pTSRow,
int64_t endKey, bool* freeTSRow) {
int32_t tsdbGetNextRowInMem(STableBlockScanInfo* pBlockScanInfo, STsdbReader* pReader, STSRow** pTSRow, int64_t endKey,
bool* freeTSRow) {
TSDBROW* pRow = getValidRow(&pBlockScanInfo->iter, pBlockScanInfo->delSkyline, pReader);
TSDBROW* piRow = getValidRow(&pBlockScanInfo->iiter, pBlockScanInfo->delSkyline, pReader);
SArray* pDelList = pBlockScanInfo->delSkyline;
......@@ -2446,7 +2448,7 @@ int32_t doAppendRowFromTSRow(SSDataBlock* pBlock, STsdbReader* pReader, STSRow*
int32_t numOfCols = (int32_t)taosArrayGetSize(pBlock->pDataBlock);
SBlockLoadSuppInfo* pSupInfo = &pReader->suppInfo;
STSchema* pSchema = doGetSchemaForTSRow(pTSRow->sver, pReader, uid);
STSchema* pSchema = doGetSchemaForTSRow(pTSRow->sver, pReader, uid);
SColVal colVal = {0};
int32_t i = 0, j = 0;
......@@ -2532,7 +2534,7 @@ int32_t buildDataBlockFromBufImpl(STableBlockScanInfo* pBlockScanInfo, int64_t e
do {
STSRow* pTSRow = NULL;
bool freeTSRow = false;
bool freeTSRow = false;
tsdbGetNextRowInMem(pBlockScanInfo, pReader, &pTSRow, endKey, &freeTSRow);
if (pTSRow == NULL) {
break;
......@@ -2581,9 +2583,7 @@ void* tsdbGetIvtIdx(SMeta* pMeta) {
return metaGetIvtIdx(pMeta);
}
uint64_t getReaderMaxVersion(STsdbReader *pReader) {
return pReader->verRange.maxVer;
}
uint64_t getReaderMaxVersion(STsdbReader* pReader) { return pReader->verRange.maxVer; }
/**
* @brief Get all suids since suid
......@@ -2761,7 +2761,8 @@ void tsdbReaderClose(STsdbReader* pReader) {
SIOCostSummary* pCost = &pReader->cost;
tsdbDebug("%p :io-cost summary: head-file:%" PRIu64 ", head-file time:%.2f ms, SMA:%" PRId64
" SMA-time:%.2f ms, fileBlocks:%" PRId64 ", fileBlocks-time:%.2f ms, "
" SMA-time:%.2f ms, fileBlocks:%" PRId64
", fileBlocks-time:%.2f ms, "
"build in-memory-block-time:%.2f ms, STableBlockScanInfo size:%.2f Kb %s",
pReader, pCost->headFileLoad, pCost->headFileLoadTime, pCost->smaData, pCost->smaLoadTime,
pCost->numOfBlocks, pCost->blockLoadTime, pCost->buildmemBlock,
......@@ -2769,7 +2770,9 @@ void tsdbReaderClose(STsdbReader* pReader) {
taosMemoryFree(pReader->idStr);
taosMemoryFree(pReader->pSchema);
taosMemoryFree(pReader->pMemSchema);
if (pReader->pMemSchema != pReader->pSchema) {
taosMemoryFree(pReader->pMemSchema);
}
taosMemoryFreeClear(pReader);
}
......
source/dnode/vnode/src/vnd/vnodeSvr.c
浏览文件 @ b924ff08
......@@ -293,6 +293,8 @@ int32_t vnodeProcessQueryMsg(SVnode *pVnode, SRpcMsg *pMsg) {
return qWorkerProcessQueryMsg(&handle, pVnode->pQuery, pMsg, 0);
case TDMT_SCH_QUERY_CONTINUE:
return qWorkerProcessCQueryMsg(&handle, pVnode->pQuery, pMsg, 0);
case TDMT_VND_FETCH_RSMA:
return smaProcessFetch(pVnode->pSma, pMsg);
default:
vError("unknown msg type:%d in query queue", pMsg->msgType);
return TSDB_CODE_VND_APP_ERROR;
......@@ -329,8 +331,6 @@ int32_t vnodeProcessFetchMsg(SVnode *pVnode, SRpcMsg *pMsg, SQueueInfo *pInfo) {
return vnodeGetTableCfg(pVnode, pMsg, true);
case TDMT_VND_BATCH_META:
return vnodeGetBatchMeta(pVnode, pMsg);
case TDMT_VND_FETCH_RSMA:
return smaProcessFetch(pVnode->pSma, pMsg);
case TDMT_VND_CONSUME:
return tqProcessPollReq(pVnode->pTq, pMsg);
case TDMT_STREAM_TASK_RUN:
......@@ -357,7 +357,7 @@ int32_t vnodeProcessFetchMsg(SVnode *pVnode, SRpcMsg *pMsg, SQueueInfo *pInfo) {
// TODO: remove the function
void smaHandleRes(void *pVnode, int64_t smaId, const SArray *data) {
// TODO
blockDebugShowDataBlocks(data, __func__);
// blockDebugShowDataBlocks(data, __func__);
tdProcessTSmaInsert(((SVnode *)pVnode)->pSma, smaId, (const char *)data);
}
......
source/libs/executor/inc/tsimplehash.h
浏览文件 @ b924ff08
......@@ -45,6 +45,8 @@ SSHashObj *tSimpleHashInit(size_t capacity, _hash_fn_t fn, size_t keyLen, size_t
*/
int32_t tSimpleHashGetSize(const SSHashObj *pHashObj);
int32_t tSimpleHashPrint(const SSHashObj *pHashObj);
/**
* put element into hash table, if the element with the same key exists, update it
* @param pHashObj
......@@ -98,6 +100,15 @@ size_t tSimpleHashGetMemSize(const SSHashObj *pHashObj);
*/
void *tSimpleHashGetKey(const SSHashObj* pHashObj, void *data, size_t* keyLen);
/**
* Create the hash table iterator
* @param pHashObj
* @param data
* @param iter
* @return void*
*/
void *tSimpleHashIterate(const SSHashObj *pHashObj, void *data, int32_t *iter);
#ifdef __cplusplus
}
#endif
......
source/libs/executor/src/executor.c
浏览文件 @ b924ff08
......@@ -529,7 +529,6 @@ int32_t qExecTask(qTaskInfo_t tinfo, SSDataBlock** pRes, uint64_t* useconds) {
cleanUpUdfs();
qDebug("%s task abort due to error/cancel occurs, code:%s", GET_TASKID(pTaskInfo), tstrerror(pTaskInfo->code));
atomic_store_64(&pTaskInfo->owner, 0);
return pTaskInfo->code;
}
......
source/libs/executor/src/executorimpl.c
浏览文件 @ b924ff08
......@@ -3612,11 +3612,16 @@ void destroyExchangeOperatorInfo(void* param, int32_t numOfOutput) {
taosRemoveRef(exchangeObjRefPool, pExInfo->self);
}
void freeSourceDataInfo(void *p) {
SSourceDataInfo* pInfo = (SSourceDataInfo*)p;
taosMemoryFreeClear(pInfo->pRsp);
}
void doDestroyExchangeOperatorInfo(void* param) {
SExchangeInfo* pExInfo = (SExchangeInfo*)param;
taosArrayDestroy(pExInfo->pSources);
taosArrayDestroy(pExInfo->pSourceDataInfo);
taosArrayDestroyEx(pExInfo->pSourceDataInfo, freeSourceDataInfo);
if (pExInfo->pResultBlockList != NULL) {
taosArrayDestroyEx(pExInfo->pResultBlockList, freeBlock);
......
source/libs/executor/src/timewindowoperator.c
浏览文件 @ b924ff08
......@@ -2215,6 +2215,7 @@ static void genInterpolationResult(STimeSliceOperatorInfo* pSliceInfo, SExprSupp
colDataAppend(pDst, rows, (char *)current.val, false);
}
taosMemoryFree(current.val);
pResBlock->info.rows += 1;
break;
}
......@@ -2437,10 +2438,10 @@ static SSDataBlock* doTimeslice(SOperatorInfo* pOperator) {
// if its the last point in data block, no need to fill, but reserve this point as the start value and do
// the interpolation when processing next data block.
if (pSliceInfo->fillType == TSDB_FILL_LINEAR) {
doKeepLinearInfo(pSliceInfo, pBlock, i, false);
pSliceInfo->current =
taosTimeAdd(pSliceInfo->current, pInterval->interval, pInterval->intervalUnit, pInterval->precision);
if (i < pBlock->info.rows - 1) {
doKeepLinearInfo(pSliceInfo, pBlock, i, false);
int64_t nextTs = *(int64_t*)colDataGetData(pTsCol, i + 1);
if (nextTs > pSliceInfo->current) {
while (pSliceInfo->current < nextTs && pSliceInfo->current <= pSliceInfo->win.ekey) {
......@@ -2478,11 +2479,11 @@ static SSDataBlock* doTimeslice(SOperatorInfo* pOperator) {
doKeepPrevRows(pSliceInfo, pBlock, i);
if (pSliceInfo->fillType == TSDB_FILL_LINEAR) {
doKeepLinearInfo(pSliceInfo, pBlock, i, false);
// no need to increate pSliceInfo->current here
//pSliceInfo->current =
// taosTimeAdd(pSliceInfo->current, pInterval->interval, pInterval->intervalUnit, pInterval->precision);
if (i < pBlock->info.rows - 1) {
doKeepLinearInfo(pSliceInfo, pBlock, i, false);
int64_t nextTs = *(int64_t*)colDataGetData(pTsCol, i + 1);
if (nextTs > pSliceInfo->current) {
while (pSliceInfo->current < nextTs && pSliceInfo->current <= pSliceInfo->win.ekey) {
......@@ -2558,10 +2559,10 @@ static SSDataBlock* doTimeslice(SOperatorInfo* pOperator) {
if (pSliceInfo->fillType == TSDB_FILL_LINEAR) {
doKeepLinearInfo(pSliceInfo, pBlock, i, false);
pSliceInfo->current =
taosTimeAdd(pSliceInfo->current, pInterval->interval, pInterval->intervalUnit, pInterval->precision);
if (i < pBlock->info.rows - 1) {
doKeepLinearInfo(pSliceInfo, pBlock, i, false);
int64_t nextTs = *(int64_t*)colDataGetData(pTsCol, i + 1);
if (nextTs > pSliceInfo->current) {
while (pSliceInfo->current < nextTs && pSliceInfo->current <= pSliceInfo->win.ekey) {
......@@ -2618,6 +2619,35 @@ static SSDataBlock* doTimeslice(SOperatorInfo* pOperator) {
return pResBlock->info.rows == 0 ? NULL : pResBlock;
}
void destroyTimeSliceOperatorInfo(void* param, int32_t numOfOutput) {
STimeSliceOperatorInfo* pInfo = (STimeSliceOperatorInfo*)param;
pInfo->pRes = blockDataDestroy(pInfo->pRes);
for (int32_t i = 0; i < taosArrayGetSize(pInfo->pPrevRow); ++i) {
SGroupKeys* pKey = taosArrayGet(pInfo->pPrevRow, i);
taosMemoryFree(pKey->pData);
}
taosArrayDestroy(pInfo->pPrevRow);
for (int32_t i = 0; i < taosArrayGetSize(pInfo->pNextRow); ++i) {
SGroupKeys* pKey = taosArrayGet(pInfo->pNextRow, i);
taosMemoryFree(pKey->pData);
}
taosArrayDestroy(pInfo->pNextRow);
for (int32_t i = 0; i < taosArrayGetSize(pInfo->pLinearInfo); ++i) {
SFillLinearInfo* pKey = taosArrayGet(pInfo->pLinearInfo, i);
taosMemoryFree(pKey->start.val);
taosMemoryFree(pKey->end.val);
}
taosArrayDestroy(pInfo->pLinearInfo);
taosMemoryFree(pInfo->pFillColInfo);
taosMemoryFreeClear(param);
}
SOperatorInfo* createTimeSliceOperatorInfo(SOperatorInfo* downstream, SPhysiNode* pPhyNode, SExecTaskInfo* pTaskInfo) {
STimeSliceOperatorInfo* pInfo = taosMemoryCalloc(1, sizeof(STimeSliceOperatorInfo));
SOperatorInfo* pOperator = taosMemoryCalloc(1, sizeof(SOperatorInfo));
......@@ -2665,7 +2695,7 @@ SOperatorInfo* createTimeSliceOperatorInfo(SOperatorInfo* downstream, SPhysiNode
pOperator->pTaskInfo = pTaskInfo;
pOperator->fpSet =
createOperatorFpSet(operatorDummyOpenFn, doTimeslice, NULL, NULL, destroyBasicOperatorInfo, NULL, NULL, NULL);
createOperatorFpSet(operatorDummyOpenFn, doTimeslice, NULL, NULL, destroyTimeSliceOperatorInfo, NULL, NULL, NULL);
blockDataEnsureCapacity(pInfo->pRes, pOperator->resultInfo.capacity);
......
source/libs/executor/src/tsimplehash.c
浏览文件 @ b924ff08
......@@ -62,7 +62,7 @@ SSHashObj *tSimpleHashInit(size_t capacity, _hash_fn_t fn, size_t keyLen, size_t
}
SSHashObj *pHashObj = (SSHashObj *)taosMemoryCalloc(1, sizeof(SSHashObj));
if (pHashObj == NULL) {
if (!pHashObj) {
terrno = TSDB_CODE_OUT_OF_MEMORY;
return NULL;
}
......@@ -78,7 +78,7 @@ SSHashObj *tSimpleHashInit(size_t capacity, _hash_fn_t fn, size_t keyLen, size_t
pHashObj->dataLen = dataLen;
pHashObj->hashList = (SHNode **)taosMemoryCalloc(pHashObj->capacity, sizeof(void *));
if (pHashObj->hashList == NULL) {
if (!pHashObj->hashList) {
taosMemoryFree(pHashObj);
terrno = TSDB_CODE_OUT_OF_MEMORY;
return NULL;
......@@ -87,7 +87,7 @@ SSHashObj *tSimpleHashInit(size_t capacity, _hash_fn_t fn, size_t keyLen, size_t
}
int32_t tSimpleHashGetSize(const SSHashObj *pHashObj) {
if (pHashObj == NULL) {
if (!pHashObj) {
return 0;
}
return (int32_t)atomic_load_64((int64_t *)&pHashObj->size);
......@@ -95,7 +95,7 @@ int32_t tSimpleHashGetSize(const SSHashObj *pHashObj) {
static SHNode *doCreateHashNode(const void *key, size_t keyLen, const void *pData, size_t dsize, uint32_t hashVal) {
SHNode *pNewNode = taosMemoryMalloc(sizeof(SHNode) + keyLen + dsize);
if (pNewNode == NULL) {
if (!pNewNode) {
terrno = TSDB_CODE_OUT_OF_MEMORY;
return NULL;
}
......@@ -120,7 +120,7 @@ static void taosHashTableResize(SSHashObj *pHashObj) {
int64_t st = taosGetTimestampUs();
void *pNewEntryList = taosMemoryRealloc(pHashObj->hashList, sizeof(void *) * newCapacity);
if (pNewEntryList == NULL) {
if (!pNewEntryList) {
// qWarn("hash resize failed due to out of memory, capacity remain:%zu", pHashObj->capacity);
return;
}
......@@ -133,22 +133,21 @@ static void taosHashTableResize(SSHashObj *pHashObj) {
for (int32_t idx = 0; idx < pHashObj->capacity; ++idx) {
SHNode *pNode = pHashObj->hashList[idx];
if (pNode == NULL) {
if (!pNode) {
continue;
}
SHNode *pNext;
SHNode *pNext = NULL;
SHNode *pPrev = NULL;
while (pNode != NULL) {
void *key = GET_SHASH_NODE_KEY(pNode, pHashObj->dataLen);
uint32_t hashVal = (*pHashObj->hashFp)(key, (uint32_t)pHashObj->dataLen);
uint32_t hashVal = (*pHashObj->hashFp)(key, (uint32_t)pHashObj->keyLen);
int32_t newIdx = HASH_INDEX(hashVal, pHashObj->capacity);
pNext = pNode->next;
if (newIdx != idx) {
if (pPrev == NULL) {
if (!pPrev) {
pHashObj->hashList[idx] = pNext;
} else {
pPrev->next = pNext;
......@@ -172,7 +171,7 @@ static void taosHashTableResize(SSHashObj *pHashObj) {
}
int32_t tSimpleHashPut(SSHashObj *pHashObj, const void *key, const void *data) {
if (pHashObj == NULL || key == NULL) {
if (!pHashObj || !key) {
return -1;
}
......@@ -186,13 +185,14 @@ int32_t tSimpleHashPut(SSHashObj *pHashObj, const void *key, const void *data) {
int32_t slot = HASH_INDEX(hashVal, pHashObj->capacity);
SHNode *pNode = pHashObj->hashList[slot];
if (pNode == NULL) {
SHNode *pNewNode = doCreateHashNode(key, pHashObj->keyLen, data, pHashObj->size, hashVal);
if (pNewNode == NULL) {
if (!pNode) {
SHNode *pNewNode = doCreateHashNode(key, pHashObj->keyLen, data, pHashObj->dataLen, hashVal);
if (!pNewNode) {
return -1;
}
pHashObj->hashList[slot] = pNewNode;
atomic_add_fetch_64(&pHashObj->size, 1);
return 0;
}
......@@ -203,9 +203,9 @@ int32_t tSimpleHashPut(SSHashObj *pHashObj, const void *key, const void *data) {
pNode = pNode->next;
}
if (pNode == NULL) {
SHNode *pNewNode = doCreateHashNode(key, pHashObj->keyLen, data, pHashObj->size, hashVal);
if (pNewNode == NULL) {
if (!pNode) {
SHNode *pNewNode = doCreateHashNode(key, pHashObj->keyLen, data, pHashObj->dataLen, hashVal);
if (!pNewNode) {
return -1;
}
pNewNode->next = pHashObj->hashList[slot];
......@@ -234,7 +234,7 @@ static FORCE_INLINE SHNode *doSearchInEntryList(SSHashObj *pHashObj, const void
static FORCE_INLINE bool taosHashTableEmpty(const SSHashObj *pHashObj) { return tSimpleHashGetSize(pHashObj) == 0; }
void *tSimpleHashGet(SSHashObj *pHashObj, const void *key) {
if (pHashObj == NULL || taosHashTableEmpty(pHashObj) || key == NULL) {
if (!pHashObj || taosHashTableEmpty(pHashObj) || !key) {
return NULL;
}
......@@ -242,7 +242,7 @@ void *tSimpleHashGet(SSHashObj *pHashObj, const void *key) {
int32_t slot = HASH_INDEX(hashVal, pHashObj->capacity);
SHNode *pNode = pHashObj->hashList[slot];
if (pNode == NULL) {
if (!pNode) {
return NULL;
}
......@@ -256,19 +256,43 @@ void *tSimpleHashGet(SSHashObj *pHashObj, const void *key) {
}
int32_t tSimpleHashRemove(SSHashObj *pHashObj, const void *key) {
// todo
if (!pHashObj || !key) {
return TSDB_CODE_FAILED;
}
uint32_t hashVal = (*pHashObj->hashFp)(key, (uint32_t)pHashObj->keyLen);
int32_t slot = HASH_INDEX(hashVal, pHashObj->capacity);
SHNode *pNode = pHashObj->hashList[slot];
SHNode *pPrev = NULL;
while (pNode) {
if ((*(pHashObj->equalFp))(GET_SHASH_NODE_KEY(pNode, pHashObj->dataLen), key, pHashObj->keyLen) == 0) {
if (!pPrev) {
pHashObj->hashList[slot] = pNode->next;
} else {
pPrev->next = pNode->next;
}
FREE_HASH_NODE(pNode);
atomic_sub_fetch_64(&pHashObj->size, 1);
break;
}
pPrev = pNode;
pNode = pNode->next;
}
return TSDB_CODE_SUCCESS;
}
void tSimpleHashClear(SSHashObj *pHashObj) {
if (pHashObj == NULL) {
if (!pHashObj || taosHashTableEmpty(pHashObj)) {
return;
}
SHNode *pNode, *pNext;
SHNode *pNode = NULL, *pNext = NULL;
for (int32_t i = 0; i < pHashObj->capacity; ++i) {
pNode = pHashObj->hashList[i];
if (pNode == NULL) {
if (!pNode) {
continue;
}
......@@ -278,11 +302,11 @@ void tSimpleHashClear(SSHashObj *pHashObj) {
pNode = pNext;
}
}
pHashObj->size = 0;
atomic_store_64(&pHashObj->size, 0);
}
void tSimpleHashCleanup(SSHashObj *pHashObj) {
if (pHashObj == NULL) {
if (!pHashObj) {
return;
}
......@@ -291,7 +315,7 @@ void tSimpleHashCleanup(SSHashObj *pHashObj) {
}
size_t tSimpleHashGetMemSize(const SSHashObj *pHashObj) {
if (pHashObj == NULL) {
if (!pHashObj) {
return 0;
}
......@@ -299,11 +323,58 @@ size_t tSimpleHashGetMemSize(const SSHashObj *pHashObj) {
}
void *tSimpleHashGetKey(const SSHashObj *pHashObj, void *data, size_t *keyLen) {
#if 0
int32_t offset = offsetof(SHNode, data);
SHNode *node = ((SHNode *)(char *)data - offset);
if (keyLen != NULL) {
if (keyLen) {
*keyLen = pHashObj->keyLen;
}
return POINTER_SHIFT(data, pHashObj->dataLen);
return GET_SHASH_NODE_KEY(node, pHashObj->dataLen);
#endif
if (keyLen) {
*keyLen = pHashObj->keyLen;
}
return POINTER_SHIFT(data, pHashObj->dataLen);
}
void *tSimpleHashIterate(const SSHashObj *pHashObj, void *data, int32_t *iter) {
if (!pHashObj) {
return NULL;
}
SHNode *pNode = NULL;
if (!data) {
for (int32_t i = 0; i < pHashObj->capacity; ++i) {
pNode = pHashObj->hashList[i];
if (!pNode) {
continue;
}
*iter = i;
return GET_SHASH_NODE_DATA(pNode);
}
return NULL;
}
pNode = (SHNode *)((char *)data - offsetof(SHNode, data));
if (pNode->next) {
return GET_SHASH_NODE_DATA(pNode->next);
}
++(*iter);
for (int32_t i = *iter; i < pHashObj->capacity; ++i) {
pNode = pHashObj->hashList[i];
if (!pNode) {
continue;
}
*iter = i;
return GET_SHASH_NODE_DATA(pNode);
}
return NULL;
}
\ No newline at end of file
source/libs/executor/test/CMakeLists.txt
浏览文件 @ b924ff08
......@@ -17,4 +17,19 @@ IF(NOT TD_DARWIN)
PUBLIC "${TD_SOURCE_DIR}/include/libs/executor/"
PRIVATE "${TD_SOURCE_DIR}/source/libs/executor/inc"
)
ENDIF ()
\ No newline at end of file
ENDIF ()
# SET(CMAKE_CXX_STANDARD 11)
# AUX_SOURCE_DIRECTORY(${CMAKE_CURRENT_SOURCE_DIR} SOURCE_LIST)
# ADD_EXECUTABLE(tSimpleHashTest tSimpleHashTests.cpp)
# TARGET_LINK_LIBRARIES(
# tSimpleHashTest
# PRIVATE os util common executor gtest_main
# )
# TARGET_INCLUDE_DIRECTORIES(
# tSimpleHashTest
# PUBLIC "${TD_SOURCE_DIR}/include/common"
# PRIVATE "${CMAKE_CURRENT_SOURCE_DIR}/../inc"
# )
\ No newline at end of file
source/libs/executor/test/tSimpleHashTests.cpp 0 → 100644
浏览文件 @ b924ff08
/*
* Copyright (c) 2019 TAOS Data, Inc. <jhtao@taosdata.com>
*
* This program is free software: you can use, redistribute, and/or modify
* it under the terms of the GNU Affero General Public License, version 3
* or later ("AGPL"), as published by the Free Software Foundation.
*
* This program is distributed in the hope that it will be useful, but WITHOUT
* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
* FITNESS FOR A PARTICULAR PURPOSE.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
#include <gtest/gtest.h>
#include <iostream>
#include "taos.h"
#include "thash.h"
#include "tsimplehash.h"
#pragma GCC diagnostic push
#pragma GCC diagnostic ignored "-Wwrite-strings"
#pragma GCC diagnostic ignored "-Wunused-function"
#pragma GCC diagnostic ignored "-Wunused-variable"
#pragma GCC diagnostic ignored "-Wsign-compare"
// int main(int argc, char **argv) {
// testing::InitGoogleTest(&argc, argv);
// return RUN_ALL_TESTS();
// }
TEST(testCase, tSimpleHashTest) {
SSHashObj *pHashObj =
tSimpleHashInit(8, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BIGINT), sizeof(int64_t), sizeof(int64_t));
assert(pHashObj != nullptr);
ASSERT_EQ(0, tSimpleHashGetSize(pHashObj));
int64_t originKeySum = 0;
for (int64_t i = 1; i <= 100; ++i) {
originKeySum += i;
tSimpleHashPut(pHashObj, (const void *)&i, (const void *)&i);
ASSERT_EQ(i, tSimpleHashGetSize(pHashObj));
}
for (int64_t i = 1; i <= 100; ++i) {
void *data = tSimpleHashGet(pHashObj, (const void *)&i);
ASSERT_EQ(i, *(int64_t *)data);
}
void *data = NULL;
int32_t iter = 0;
int64_t keySum = 0;
int64_t dataSum = 0;
while ((data = tSimpleHashIterate(pHashObj, data, &iter))) {
void *key = tSimpleHashGetKey(pHashObj, data, NULL);
keySum += *(int64_t *)key;
dataSum += *(int64_t *)data;
}
ASSERT_EQ(keySum, dataSum);
ASSERT_EQ(keySum, originKeySum);
for (int64_t i = 1; i <= 100; ++i) {
tSimpleHashRemove(pHashObj, (const void *)&i);
ASSERT_EQ(100 - i, tSimpleHashGetSize(pHashObj));
}
tSimpleHashCleanup(pHashObj);
}
#pragma GCC diagnostic pop
\ No newline at end of file
source/libs/parser/src/parTranslater.c
浏览文件 @ b924ff08
......@@ -1350,6 +1350,7 @@ static void setFuncClassification(SNode* pCurrStmt, SFunctionNode* pFunc) {
pSelect->hasInterpFunc = pSelect->hasInterpFunc ? true : (FUNCTION_TYPE_INTERP == pFunc->funcType);
pSelect->hasLastRowFunc = pSelect->hasLastRowFunc ? true : (FUNCTION_TYPE_LAST_ROW == pFunc->funcType);
pSelect->hasTimeLineFunc = pSelect->hasTimeLineFunc ? true : fmIsTimelineFunc(pFunc->funcId);
pSelect->hasUdaf = pSelect->hasUdaf ? true : fmIsUserDefinedFunc(pFunc->funcId) && fmIsAggFunc(pFunc->funcId);
pSelect->onlyHasKeepOrderFunc = pSelect->onlyHasKeepOrderFunc ? fmIsKeepOrderFunc(pFunc->funcId) : false;
}
}
......@@ -2644,6 +2645,11 @@ static int32_t translateInterp(STranslateContext* pCxt, SSelectStmt* pSelect) {
return TSDB_CODE_SUCCESS;
}
if (NULL == pSelect->pRange || NULL == pSelect->pEvery || NULL == pSelect->pFill) {
return generateSyntaxErrMsgExt(&pCxt->msgBuf, TSDB_CODE_PAR_INVALID_INTERP_CLAUSE,
"Missing RANGE clause, EVERY clause or FILL clause");
}
int32_t code = translateExpr(pCxt, &pSelect->pRange);
if (TSDB_CODE_SUCCESS == code) {
code = translateExpr(pCxt, &pSelect->pEvery);
......@@ -4734,6 +4740,11 @@ static bool crossTableWithoutAggOper(SSelectStmt* pSelect) {
!isPartitionByTbname(pSelect->pPartitionByList);
}
static bool crossTableWithUdaf(SSelectStmt* pSelect) {
return pSelect->hasUdaf && TSDB_SUPER_TABLE == ((SRealTableNode*)pSelect->pFromTable)->pMeta->tableType &&
!isPartitionByTbname(pSelect->pPartitionByList);
}
static int32_t checkCreateStream(STranslateContext* pCxt, SCreateStreamStmt* pStmt) {
if (NULL != pStmt->pOptions->pWatermark &&
(DEAL_RES_ERROR == translateValue(pCxt, (SValueNode*)pStmt->pOptions->pWatermark))) {
......@@ -4785,7 +4796,8 @@ static int32_t addWstartTsToCreateStreamQuery(SNode* pStmt) {
static int32_t checkStreamQuery(STranslateContext* pCxt, SSelectStmt* pSelect) {
if (TSDB_DATA_TYPE_TIMESTAMP != ((SExprNode*)nodesListGetNode(pSelect->pProjectionList, 0))->resType.type ||
!pSelect->isTimeLineResult || crossTableWithoutAggOper(pSelect)) {
!pSelect->isTimeLineResult || crossTableWithoutAggOper(pSelect) || NULL != pSelect->pOrderByList ||
crossTableWithUdaf(pSelect)) {
return generateSyntaxErrMsgExt(&pCxt->msgBuf, TSDB_CODE_PAR_INVALID_STREAM_QUERY, "Unsupported stream query");
}
return TSDB_CODE_SUCCESS;
......
source/libs/parser/test/parSelectTest.cpp
浏览文件 @ b924ff08
......@@ -294,16 +294,6 @@ TEST_F(ParserSelectTest, intervalSemanticCheck) {
TEST_F(ParserSelectTest, interp) {
useDb("root", "test");
run("SELECT INTERP(c1) FROM t1");
run("SELECT INTERP(c1) FROM t1 RANGE('2017-7-14 18:00:00', '2017-7-14 19:00:00')");
run("SELECT INTERP(c1) FROM t1 RANGE('2017-7-14 18:00:00', '2017-7-14 19:00:00') FILL(LINEAR)");
run("SELECT INTERP(c1) FROM t1 EVERY(5s)");
run("SELECT INTERP(c1) FROM t1 RANGE('2017-7-14 18:00:00', '2017-7-14 19:00:00') EVERY(5s)");
run("SELECT INTERP(c1) FROM t1 RANGE('2017-7-14 18:00:00', '2017-7-14 19:00:00') EVERY(5s) FILL(LINEAR)");
}
......
source/libs/planner/src/planOptimizer.c
浏览文件 @ b924ff08
......@@ -1615,6 +1615,9 @@ static int32_t partTagsOptimize(SOptimizeContext* pCxt, SLogicSubplan* pLogicSub
if (QUERY_NODE_LOGIC_PLAN_PARTITION == nodeType(pNode)) {
TSWAP(((SPartitionLogicNode*)pNode)->pPartitionKeys, pScan->pGroupTags);
int32_t code = replaceLogicNode(pLogicSubplan, pNode, (SLogicNode*)pScan);
if (TSDB_CODE_SUCCESS == code) {
code = adjustLogicNodeDataRequirement((SLogicNode*)pScan, pNode->resultDataOrder);
}
if (TSDB_CODE_SUCCESS == code) {
NODES_CLEAR_LIST(pNode->pChildren);
nodesDestroyNode((SNode*)pNode);
......
source/libs/planner/test/planBasicTest.cpp
浏览文件 @ b924ff08
......@@ -93,8 +93,6 @@ TEST_F(PlanBasicTest, tailFunc) {
TEST_F(PlanBasicTest, interpFunc) {
useDb("root", "test");
run("SELECT INTERP(c1) FROM t1");
run("SELECT INTERP(c1) FROM t1 RANGE('2017-7-14 18:00:00', '2017-7-14 19:00:00') EVERY(5s) FILL(LINEAR)");
}
......
tests/script/tsim/sma/tsmaCreateInsertQuery.sim
浏览文件 @ b924ff08
......@@ -61,6 +61,7 @@ endi
print =============== select * from stb from memory in designated vgroup
sql select _wstart, _wend, min(c1),max(c2),max(c1) from stb interval(5m,10s) sliding(5m);
print $data00 $data01 $data02 $data03 $data04
print $data10 $data11 $data12 $data13 $data14
if $rows != 1 then
print rows $rows != 1
return -1
......
tests/system-test/2-query/Now.py
浏览文件 @ b924ff08
......@@ -9,12 +9,13 @@ class TDTestCase:
def init(self, conn, logSql):
tdLog.debug(f"start to excute {__file__}")
tdSql.init(conn.cursor(),True)
tdSql.init(conn.cursor())
self.setsql = TDSetSql()
self.dbname = 'db'
# name of normal table
self.ntbname = 'ntb'
self.ntbname = f'{self.dbname}.ntb'
# name of stable
self.stbname = 'stb'
self.stbname = f'{self.dbname}.stb'
# structure of column
self.column_dict = {
'ts':'timestamp',
......@@ -72,19 +73,19 @@ class TDTestCase:
def now_check_ntb(self):
for time_unit in self.db_percision:
tdSql.execute(f'create database db precision "{time_unit}"')
tdSql.execute('use db')
tdSql.execute(f'create database {self.dbname} precision "{time_unit}"')
tdSql.execute(f'use {self.dbname}')
tdSql.execute(self.setsql.set_create_normaltable_sql(self.ntbname,self.column_dict))
for value in self.values_list:
tdSql.execute(
f'insert into {self.ntbname} values({value})')
self.data_check(self.ntbname,'normal table')
tdSql.execute('drop database db')
tdSql.execute(f'drop database {self.dbname}')
def now_check_stb(self):
for time_unit in self.db_percision:
tdSql.execute(f'create database db precision "{time_unit}"')
tdSql.execute('use db')
tdSql.execute(f'create database {self.dbname} precision "{time_unit}"')
tdSql.execute(f'use {self.dbname}')
tdSql.execute(self.setsql.set_create_stable_sql(self.stbname,self.column_dict,self.tag_dict))
for i in range(self.tbnum):
tdSql.execute(f"create table {self.stbname}_{i} using {self.stbname} tags({self.tag_values[0]})")
......@@ -93,7 +94,7 @@ class TDTestCase:
for i in range(self.tbnum):
self.data_check(f'{self.stbname}_{i}','child table')
self.data_check(self.stbname,'stable')
tdSql.execute('drop database db')
tdSql.execute(f'drop database {self.dbname}')
def run(self): # sourcery skip: extract-duplicate-method
self.now_check_ntb()
......
tests/system-test/2-query/Timediff.py
浏览文件 @ b924ff08
......@@ -19,9 +19,10 @@ class TDTestCase:
self.db_param_precision = ['ms','us','ns']
self.time_unit = ['1w','1d','1h','1m','1s','1a','1u','1b']
self.error_unit = ['2w','2d','2h','2m','2s','2a','2u','1c','#1']
self.ntbname = 'ntb'
self.stbname = 'stb'
self.ctbname = 'ctb'
self.dbname = 'db'
self.ntbname = f'{self.dbname}.ntb'
self.stbname = f'{self.dbname}.stb'
self.ctbname = f'{self.dbname}.ctb'
self.subtractor = 1 # unit:s
def check_tbtype(self,tb_type):
if tb_type.lower() == 'ntb':
......@@ -139,9 +140,9 @@ class TDTestCase:
tdSql.error(f'select timediff(c0,{self.subtractor},{unit}) from {self.ntbname}')
def function_check_ntb(self):
for precision in self.db_param_precision:
tdSql.execute('drop database if exists db')
tdSql.execute(f'create database db precision "{precision}"')
tdSql.execute('use db')
tdSql.execute(f'drop database if exists {self.dbname}')
tdSql.execute(f'create database {self.dbname} precision "{precision}"')
tdSql.execute(f'use {self.dbname}')
tdSql.execute(f'create table {self.ntbname} (ts timestamp,c0 int)')
for ts in self.ts_str:
tdSql.execute(f'insert into {self.ntbname} values("{ts}",1)')
......@@ -151,9 +152,9 @@ class TDTestCase:
self.data_check(date_time,precision,'ntb')
def function_check_stb(self):
for precision in self.db_param_precision:
tdSql.execute('drop database if exists db')
tdSql.execute(f'create database db precision "{precision}"')
tdSql.execute('use db')
tdSql.execute(f'drop database if exists {self.dbname}')
tdSql.execute(f'create database {self.dbname} precision "{precision}"')
tdSql.execute(f'use {self.dbname}')
tdSql.execute(f'create table {self.stbname} (ts timestamp,c0 int) tags(t0 int)')
tdSql.execute(f'create table {self.ctbname} using {self.stbname} tags(1)')
for ts in self.ts_str:
......
tests/system-test/2-query/To_iso8601.py
浏览文件 @ b924ff08
......@@ -15,19 +15,20 @@ class TDTestCase:
tdSql.init(conn.cursor())
self.rowNum = 10
self.ts = 1640966400000 # 2022-1-1 00:00:00.000
self.dbname = 'db'
self.stbname = f'{self.dbname}.stb'
self.ntbname = f'{self.dbname}.ntb'
def check_customize_param_ms(self):
time_zone = time.strftime('%z')
tdSql.execute('create database db1 precision "ms"')
tdSql.execute('use db1')
tdSql.execute('create table if not exists ntb(ts timestamp, c1 int, c2 timestamp)')
tdSql.execute(f'drop database if exists {self.dbname}')
tdSql.execute(f'create database {self.dbname} precision "ms"')
tdSql.execute(f'use {self.dbname}')
tdSql.execute(f'create table if not exists {self.ntbname}(ts timestamp, c1 int, c2 timestamp)')
for i in range(self.rowNum):
tdSql.execute("insert into ntb values(%d, %d, %d)"
% (self.ts + i, i + 1, self.ts + i))
tdSql.query('select to_iso8601(ts) from ntb')
tdSql.execute(f"insert into {self.ntbname} values({self.ts + i}, {i + 1}, {self.ts + i})")
tdSql.query(f'select to_iso8601(ts) from {self.ntbname}')
for i in range(self.rowNum):
tdSql.checkEqual(tdSql.queryResult[i][0],f'2022-01-01T00:00:00.00{i}{time_zone}')
timezone_list = ['+0000','+0100','+0200','+0300','+0330','+0400','+0500','+0530','+0600','+0700','+0800','+0900','+1000','+1100','+1200',\
'+00','+01','+02','+03','+04','+05','+06','+07','+08','+09','+10','+11','+12',\
'+00:00','+01:00','+02:00','+03:00','+03:30','+04:00','+05:00','+05:30','+06:00','+07:00','+08:00','+09:00','+10:00','+11:00','+12:00',\
......@@ -36,62 +37,49 @@ class TDTestCase:
'-00:00','-01:00','-02:00','-03:00','-04:00','-05:00','-06:00','-07:00','-08:00','-09:00','-10:00','-11:00','-12:00',\
'z','Z']
for j in timezone_list:
tdSql.query(f'select to_iso8601(ts,"{j}") from ntb')
tdSql.query(f'select to_iso8601(ts,"{j}") from {self.ntbname}')
for i in range(self.rowNum):
tdSql.checkEqual(tdSql.queryResult[i][0],f'2022-01-01T00:00:00.00{i}{j}')
error_param_list = [0,100.5,'a','!']
for i in error_param_list:
tdSql.error(f'select to_iso8601(ts,"{i}") from ntb')
#! bug TD-16372:对于错误的时区，缺少校验
tdSql.error(f'select to_iso8601(ts,"{i}") from {self.ntbname}')
error_timezone_param = ['+13','-13','+1300','-1300','+0001','-0001','-0330','-0530']
for i in error_timezone_param:
tdSql.error(f'select to_iso8601(ts,"{i}") from ntb')
tdSql.error(f'select to_iso8601(ts,"{i}") from {self.ntbname}')
def check_base_function(self):
tdSql.prepare()
tdLog.printNoPrefix("==========step1:create tables==========")
tdSql.execute('create table if not exists ntb(ts timestamp, c1 int, c2 float,c3 double,c4 timestamp)')
tdSql.execute('create table if not exists stb(ts timestamp, c1 int, c2 float,c3 double,c4 timestamp) tags(t0 int)')
tdSql.execute('create table if not exists stb_1 using stb tags(100)')
tdLog.printNoPrefix("==========step2:insert data==========")
tdSql.execute('insert into ntb values(now,1,1.55,100.555555,today())("2020-1-1 00:00:00",10,11.11,99.999999,now())(today(),3,3.333,333.333333,now())')
tdSql.execute('insert into stb_1 values(now,1,1.55,100.555555,today())("2020-1-1 00:00:00",10,11.11,99.999999,now())(today(),3,3.333,333.333333,now())')
tdSql.query("select to_iso8601(ts) from ntb")
tdSql.execute('create table if not exists db.ntb(ts timestamp, c1 int, c2 float,c3 double,c4 timestamp)')
tdSql.execute('create table if not exists db.stb(ts timestamp, c1 int, c2 float,c3 double,c4 timestamp) tags(t0 int)')
tdSql.execute('create table if not exists db.stb_1 using db.stb tags(100)')
tdSql.execute('insert into db.ntb values(now,1,1.55,100.555555,today())("2020-1-1 00:00:00",10,11.11,99.999999,now())(today(),3,3.333,333.333333,now())')
tdSql.execute('insert into db.stb_1 values(now,1,1.55,100.555555,today())("2020-1-1 00:00:00",10,11.11,99.999999,now())(today(),3,3.333,333.333333,now())')
tdSql.query("select to_iso8601(ts) from db.ntb")
tdSql.checkRows(3)
tdSql.query("select c1 from ntb where ts = to_iso8601(1577808000000)")
tdSql.query("select c1 from db.ntb where ts = to_iso8601(1577808000000)")
tdSql.checkRows(1)
tdSql.checkData(0,0,10)
tdSql.query("select * from ntb where ts = to_iso8601(1577808000000)")
tdSql.query("select * from db.ntb where ts = to_iso8601(1577808000000)")
tdSql.checkRows(1)
tdSql.query("select to_iso8601(ts) from ntb where ts=today()")
tdSql.query("select to_iso8601(ts) from db.ntb where ts=today()")
tdSql.checkRows(1)
for i in range(0,3):
tdSql.query("select to_iso8601(1) from ntb")
tdSql.query("select to_iso8601(1) from db.ntb")
tdSql.checkData(i,0,"1970-01-01T08:00:01+0800")
tdSql.checkRows(3)
tdSql.query("select to_iso8601(ts) from ntb")
tdSql.checkRows(3)
tdSql.query("select to_iso8601(ts) from db.ntb")
tdSql.query("select to_iso8601(today()) from ntb")
tdSql.checkRows(3)
tdSql.query("select to_iso8601(now()) from ntb")
tdSql.query("select to_iso8601(today()) from db.ntb")
tdSql.checkRows(3)
tdSql.error("select to_iso8601(timezone()) from ntb")
tdSql.error("select to_iso8601('abc') from ntb")
tdSql.query("select to_iso8601(now()) from db.ntb")
tdSql.checkRows(3)
tdSql.error("select to_iso8601(timezone()) from db.ntb")
tdSql.error("select to_iso8601('abc') from db.ntb")
for i in ['+','-','*','/']:
tdSql.query(f"select to_iso8601(today()) {i}null from ntb")
tdSql.checkRows(3)
tdSql.checkData(0,0,None)
tdSql.query(f"select to_iso8601(today()) {i}null from db.ntb")
tdSql.checkRows(3)
tdSql.checkData(0,0,None)
tdSql.query("select to_iso8601(9223372036854775807) from ntb")
tdSql.query("select to_iso8601(9223372036854775807) from db.ntb")
tdSql.checkRows(3)
# bug TD-15207
# tdSql.query("select to_iso8601(10000000000) from ntb")
......@@ -102,27 +90,22 @@ class TDTestCase:
# tdSql.checkData(0,0,None)
err_param = [1.5,'a','c2']
for i in err_param:
tdSql.error(f"select to_iso8601({i}) from ntb")
tdSql.error(f"select to_iso8601({i}) from db.ntb")
tdSql.query("select to_iso8601(now) from stb")
tdSql.query("select to_iso8601(now) from db.stb")
tdSql.checkRows(3)
tdSql.query("select to_iso8601(now()) from stb")
tdSql.query("select to_iso8601(now()) from db.stb")
tdSql.checkRows(3)
tdSql.query("select to_iso8601(1) from stb")
tdSql.query("select to_iso8601(1) from db.stb")
for i in range(0,3):
tdSql.checkData(i,0,"1970-01-01T08:00:01+0800")
tdSql.checkRows(3)
tdSql.query("select to_iso8601(ts) from stb")
tdSql.query("select to_iso8601(ts) from db.stb")
tdSql.checkRows(3)
tdSql.query("select to_iso8601(ts)+1 from stb")
tdSql.query("select to_iso8601(ts)+1 from db.stb")
tdSql.checkRows(3)
tdSql.query("select to_iso8601(ts)+'a' from stb ")
tdSql.query("select to_iso8601(ts)+'a' from db.stb ")
tdSql.checkRows(3)
for i in ['+','-','*','/']:
tdSql.query(f"select to_iso8601(today()) {i}null from stb")
tdSql.checkRows(3)
tdSql.checkData(0,0,None)
tdSql.query(f"select to_iso8601(today()) {i}null from db.stb")
tdSql.checkRows(3)
tdSql.checkData(0,0,None)
......
tests/system-test/2-query/To_unixtimestamp.py
浏览文件 @ b924ff08
......@@ -3,6 +3,7 @@ from time import sleep
from util.log import *
from util.sql import *
from util.cases import *
from util.sqlset import TDSetSql
......@@ -12,17 +13,19 @@ class TDTestCase:
def init(self, conn, logSql):
tdLog.debug(f"start to excute {__file__}")
tdSql.init(conn.cursor())
self.setsql = TDSetSql()
self.dbname = 'db'
# name of normal table
self.ntbname = 'ntb'
self.ntbname = f'{self.dbname}.ntb'
# name of stable
self.stbname = 'stb'
self.stbname = f'{self.dbname}.stb'
# structure of column
self.column_dict = {
'ts':'timestamp',
'c1':'int',
'c2':'float',
'c3':'binary(20)',
'c4':'nchar(20)'
'c3':'binary(20)'
}
# structure of tag
self.tag_dict = {
......@@ -32,69 +35,68 @@ class TDTestCase:
self.tbnum = 2
# values of tag,the number of values should equal to tbnum
self.tag_values = [
f'10',
f'100'
'10',
'100'
]
# values of rows, structure should be same as column
self.values_list = [
f'now,10,99.99,"2020-1-1 00:00:00"',
f'today(),100,11.111,22.222222'
f'now,10,99.99,"abc"',
f'today(),100,11.111,"abc"'
]
self.error_param = [1,'now()']
self.error_param = [1,1.5,'now()']
def data_check(self,tbname,values_list,tb_type,tb_num=1):
for time in ['1970-01-01T08:00:00+0800','1970-01-01T08:00:00+08:00']:
tdSql.query(f"select to_unixtimestamp('{time}') from {tbname}")
if tb_type == 'ntb' or tb_type == 'ctb':
tdSql.checkRows(len(values_list))
for i in range(len(values_list)):
tdSql.checkEqual(tdSql.queryResult[i][0],0)
elif tb_type == 'stb':
tdSql.checkRows(len(self.values_list)*tb_num)
for time in ['1900-01-01T08:00:00+08:00']:
tdSql.query(f"select to_unixtimestamp('{time}') from {tbname}")
if tb_type == 'ntb' or tb_type == 'ctb':
tdSql.checkRows(len(values_list))
elif tb_type == 'stb':
tdSql.checkRows(len(self.values_list)*tb_num)
for time in ['2020-01-32T08:00:00','2020-13-32T08:00:00','acd']:
tdSql.query(f"select to_unixtimestamp('{time}') from {tbname}")
if tb_type == 'ntb' or tb_type == 'ctb':
tdSql.checkRows(len(values_list))
for i in range(len(values_list)):
tdSql.checkEqual(tdSql.queryResult[i][0],None)
elif tb_type == 'stb':
tdSql.checkRows(len(values_list)*tb_num)
for i in self.column_dict.keys():
tdSql.query(f"select {i} from {tbname} where to_unixtimestamp('1970-01-01T08:00:00+08:00')=0")
if tb_type == 'ntb' or tb_type == 'ctb':
tdSql.checkRows(len(values_list))
elif tb_type == 'stb':
tdSql.checkRows(len(values_list)*tb_num)
for time in self.error_param:
tdSql.error(f"select to_unixtimestamp({time}) from {tbname}")
def timestamp_change_check_ntb(self):
tdSql.execute(f'create database {self.dbname}')
tdSql.execute(self.setsql.set_create_normaltable_sql(self.ntbname,self.column_dict))
for i in range(len(self.values_list)):
tdSql.execute(f'insert into {self.ntbname} values({self.values_list[i]})')
self.data_check(self.ntbname,self.values_list,'ntb')
tdSql.execute(f'drop database {self.dbname}')
def timestamp_change_check_stb(self):
tdSql.execute(f'create database {self.dbname}')
tdSql.execute(self.setsql.set_create_stable_sql(self.stbname,self.column_dict,self.tag_dict))
for i in range(self.tbnum):
tdSql.execute(f'create table {self.stbname}_{i} using {self.stbname} tags({self.tag_values[i]})')
for j in range(len(self.values_list)):
tdSql.execute(f'insert into {self.stbname}_{i} values({self.values_list[j]})')
for i in range(self.tbnum):
self.data_check(f'{self.stbname}_{i}',self.values_list,'ctb')
self.data_check(self.stbname,self.values_list,'stb',self.tbnum)
tdSql.execute(f'drop database {self.dbname}')
def run(self): # sourcery skip: extract-duplicate-method
tdSql.prepare()
tdLog.printNoPrefix("==========step1:create tables==========")
tdSql.execute(
'''create table if not exists ntb
(ts timestamp, c1 int, c2 float,c3 double,c4 timestamp)
'''
)
tdSql.execute(
'''create table if not exists stb
(ts timestamp, c1 int, c2 float,c3 double,c4 timestamp) tags(t0 int)
'''
)
tdSql.execute(
'''create table if not exists stb_1 using stb tags(100)
'''
)
tdLog.printNoPrefix("==========step2:insert data into ntb==========")
# RFC3339:2020-01-01T00:00:00+8:00
# ISO8601:2020-01-01T00:00:00.000+0800
tdSql.execute(
'insert into ntb values(now,1,1.55,100.555555,today())("2020-1-1 00:00:00",10,11.11,99.999999,now())(today(),3,3.333,333.333333,now())')
tdSql.execute(
'insert into stb_1 values(now,1,1.55,100.555555,today())("2020-1-1 00:00:00",10,11.11,99.999999,now())(today(),3,3.333,333.333333,now())')
tdSql.query("select to_unixtimestamp('1970-01-01T08:00:00+0800') from ntb")
tdSql.checkData(0,0,0)
tdSql.checkData(1,0,0)
tdSql.checkData(2,0,0)
tdSql.checkRows(3)
tdSql.query("select to_unixtimestamp('1970-01-01T08:00:00+08:00') from ntb")
tdSql.checkData(0,0,0)
tdSql.checkRows(3)
tdSql.query("select to_unixtimestamp('1900-01-01T08:00:00+08:00') from ntb")
tdSql.checkRows(3)
tdSql.query("select to_unixtimestamp('2020-01-32T08:00:00') from ntb")
tdSql.checkRows(3)
tdSql.checkData(0,0,None)
tdSql.query("select to_unixtimestamp('2020-13-32T08:00:00') from ntb")
tdSql.checkRows(3)
tdSql.checkData(0,0,None)
tdSql.query("select to_unixtimestamp('acd') from ntb")
tdSql.checkRows(3)
tdSql.checkData(0,0,None)
tdSql.error("select to_unixtimestamp(1) from ntb")
tdSql.error("select to_unixtimestamp(1.5) from ntb")
tdSql.error("select to_unixtimestamp(ts) from ntb")
tdSql.query("select ts from ntb where to_unixtimestamp('1970-01-01T08:00:00+08:00')=0")
tdSql.checkRows(3)
self.timestamp_change_check_ntb()
self.timestamp_change_check_stb()
def stop(self):
tdSql.close()
tdLog.success(f"{__file__} successfully executed")
......
tests/system-test/2-query/Today.py
浏览文件 @ b924ff08
......@@ -20,8 +20,9 @@ class TDTestCase:
self.arithmetic_operators = ['+','-','*','/']
self.relational_operator = ['<','<=','=','>=','>']
# prepare data
self.ntbname = 'ntb'
self.stbname = 'stb'
self.dbname = 'db'
self.ntbname = f'{self.dbname}.ntb'
self.stbname = f'{self.dbname}.stb'
self.column_dict = {
'ts':'timestamp',
'c1':'int',
......@@ -96,7 +97,7 @@ class TDTestCase:
tdSql.checkRows(num_same*tb_num)
elif tb == 'stb':
tdSql.checkRows(num_same)
for i in [f'{tbname}',f'db.{tbname}']:
for i in [f'{tbname}']:
for unit in self.time_unit:
for symbol in ['+','-']:
tdSql.query(f"select today() {symbol}1{unit} from {i}")
......@@ -148,20 +149,20 @@ class TDTestCase:
tdSql.checkData(i, 0, str(self.today_date))
def today_check_ntb(self):
for time_unit in self.db_percision:
print(time_unit)
tdSql.execute(f'create database db precision "{time_unit}"')
tdSql.execute('use db')
tdSql.execute(f'create database {self.dbname} precision "{time_unit}"')
tdSql.execute(f'use {self.dbname}')
tdSql.execute(self.set_create_normaltable_sql(self.ntbname,self.column_dict))
for i in self.values_list:
tdSql.execute(
f'insert into {self.ntbname} values({i})')
self.data_check(self.column_dict,self.ntbname,self.values_list,1,'tb',time_unit)
tdSql.execute('drop database db')
tdSql.execute(f'drop database {self.dbname}')
def today_check_stb_tb(self):
for time_unit in self.db_percision:
print(time_unit)
tdSql.execute(f'create database db precision "{time_unit}"')
tdSql.execute('use db')
tdSql.execute(f'create database {self.dbname} precision "{time_unit}"')
tdSql.execute(f'use {self.dbname}')
tdSql.execute(self.set_create_stable_sql(self.stbname,self.column_dict,self.tag_dict))
for i in range(self.tbnum):
tdSql.execute(f'create table if not exists {self.stbname}_{i} using {self.stbname} tags({self.tag_values[i]})')
......@@ -172,7 +173,7 @@ class TDTestCase:
self.data_check(self.column_dict,f'{self.stbname}_{i}',self.values_list,1,'tb',time_unit)
# check stable
self.data_check(self.column_dict,self.stbname,self.values_list,self.tbnum,'stb',time_unit)
tdSql.execute('drop database db')
tdSql.execute(f'drop database {self.dbname}')
def run(self): # sourcery skip: extract-duplicate-method
......
tests/system-test/2-query/last.py
浏览文件 @ b924ff08
......@@ -37,7 +37,7 @@ class TDTestCase:
def last_check_stb_tb_base(self):
tdSql.prepare()
stbname = tdCom.getLongName(5, "letters")
stbname = f'db.{tdCom.getLongName(5, "letters")}'
column_dict = {
'col1': 'tinyint',
'col2': 'smallint',
......@@ -61,7 +61,7 @@ class TDTestCase:
tdSql.execute(f"create table {stbname}_1 using {stbname} tags('beijing')")
tdSql.execute(f"insert into {stbname}_1(ts) values(%d)" % (self.ts - 1))
for i in [f'{stbname}_1', f'db.{stbname}_1']:
for i in [f'{stbname}_1']:
tdSql.query(f"select last(*) from {i}")
tdSql.checkRows(1)
tdSql.checkData(0, 1, None)
......@@ -71,7 +71,7 @@ class TDTestCase:
# tdSql.checkRows(1)
# tdSql.checkData(0, 1, None)
for i in column_dict.keys():
for j in [f'{stbname}_1', f'db.{stbname}_1', f'{stbname}', f'db.{stbname}']:
for j in [f'{stbname}_1', f'{stbname}']:
tdSql.query(f"select last({i}) from {j}")
tdSql.checkRows(0)
tdSql.query(f"select last({list(column_dict.keys())[0]}) from {stbname}_1 group by {list(column_dict.keys())[-1]}")
......@@ -79,12 +79,12 @@ class TDTestCase:
for i in range(self.rowNum):
tdSql.execute(f"insert into {stbname}_1 values(%d, %d, %d, %d, %d, %d, %d, %d, %d, %f, %f, %d, '{self.binary_str}%d', '{self.nchar_str}%d')"
% (self.ts + i, i + 1, i + 1, i + 1, i + 1, i + 1, i + 1, i + 1, i + 1, i + 0.1, i + 0.1, i % 2, i + 1, i + 1))
for i in [f'{stbname}_1', f'db.{stbname}_1', f'{stbname}', f'db.{stbname}']:
for i in [f'{stbname}_1',f'{stbname}']:
tdSql.query(f"select last(*) from {i}")
tdSql.checkRows(1)
tdSql.checkData(0, 1, 10)
for k, v in column_dict.items():
for j in [f'{stbname}_1', f'db.{stbname}_1', f'{stbname}', f'db.{stbname}']:
for j in [f'{stbname}_1', f'{stbname}']:
tdSql.query(f"select last({k}) from {j}")
tdSql.checkRows(1)
# tinyint,smallint,int,bigint,tinyint unsigned,smallint unsigned,int unsigned,bigint unsigned
......@@ -103,7 +103,7 @@ class TDTestCase:
# nchar
elif 'nchar' in v.lower():
tdSql.checkData(0, 0, f'{self.nchar_str}{self.rowNum}')
for i in [f'{stbname}_1', f'db.{stbname}_1', f'{stbname}', f'db.{stbname}']:
for i in [f'{stbname}_1', f'{stbname}']:
tdSql.query(f"select last({list(column_dict.keys())[0]},{list(column_dict.keys())[1]},{list(column_dict.keys())[2]}) from {stbname}_1")
tdSql.checkData(0, 2, 10)
......@@ -113,7 +113,7 @@ class TDTestCase:
def last_check_ntb_base(self):
tdSql.prepare()
ntbname = tdCom.getLongName(5, "letters")
ntbname = f'db.{tdCom.getLongName(5, "letters")}'
column_dict = {
'col1': 'tinyint',
'col2': 'smallint',
......@@ -135,11 +135,8 @@ class TDTestCase:
tdSql.query(f"select last(*) from {ntbname}")
tdSql.checkRows(1)
tdSql.checkData(0, 1, None)
tdSql.query(f"select last(*) from db.{ntbname}")
tdSql.checkRows(1)
tdSql.checkData(0, 1, None)
for i in column_dict.keys():
for j in [f'{ntbname}', f'db.{ntbname}']:
for j in [f'{ntbname}']:
tdSql.query(f"select last({i}) from {j}")
tdSql.checkRows(0)
for i in range(self.rowNum):
......@@ -148,11 +145,8 @@ class TDTestCase:
tdSql.query(f"select last(*) from {ntbname}")
tdSql.checkRows(1)
tdSql.checkData(0, 1, 10)
tdSql.query(f"select last(*) from db.{ntbname}")
tdSql.checkRows(1)
tdSql.checkData(0, 1, 10)
for k, v in column_dict.items():
for j in [f'{ntbname}', f'db.{ntbname}']:
for j in [f'{ntbname}']:
tdSql.query(f"select last({k}) from {j}")
tdSql.checkRows(1)
# tinyint,smallint,int,bigint,tinyint unsigned,smallint unsigned,int unsigned,bigint unsigned
......@@ -178,8 +172,8 @@ class TDTestCase:
def last_check_stb_distribute(self):
# prepare data for vgroup 4
dbname = tdCom.getLongName(10, "letters")
stbname = tdCom.getLongName(5, "letters")
vgroup_num = 4
stbname = f'{dbname}.{tdCom.getLongName(5, "letters")}'
vgroup_num = 2
column_dict = {
'col1': 'tinyint',
'col2': 'smallint',
......@@ -208,11 +202,7 @@ class TDTestCase:
f"create table {stbname}_{i} using {stbname} tags('beijing')")
tdSql.execute(
f"insert into {stbname}_{i}(ts) values(%d)" % (self.ts - 1-i))
# for i in [f'{stbname}', f'{dbname}.{stbname}']:
# tdSql.query(f"select last(*) from {i}")
# tdSql.checkRows(1)
# tdSql.checkData(0, 1, None)
tdSql.query('show tables')
tdSql.query(f'show {dbname}.tables')
vgroup_list = []
for i in range(len(tdSql.queryResult)):
vgroup_list.append(tdSql.queryResult[i][6])
......@@ -222,20 +212,17 @@ class TDTestCase:
if vgroups_num >= 2:
tdLog.info(f'This scene with {vgroups_num} vgroups is ok!')
continue
# else:
# tdLog.exit(
# f'This scene does not meet the requirements with {vgroups_num} vgroup!\n')
for i in range(self.tbnum):
for j in range(self.rowNum):
tdSql.execute(f"insert into {stbname}_{i} values(%d, %d, %d, %d, %d, %d, %d, %d, %d, %f, %f, %d, '{self.binary_str}%d', '{self.nchar_str}%d')"
% (self.ts + j + i, j + 1, j + 1, j + 1, j + 1, j + 1, j + 1, j + 1, j + 1, j + 0.1, j + 0.1, j % 2, j + 1, j + 1))
for i in [f'{stbname}', f'{dbname}.{stbname}']:
for i in [f'{stbname}']:
tdSql.query(f"select last(*) from {i}")
tdSql.checkRows(1)
tdSql.checkData(0, 1, 10)
for k, v in column_dict.items():
for j in [f'{stbname}', f'{dbname}.{stbname}']:
for j in [f'{stbname}']:
tdSql.query(f"select last({k}) from {j}")
tdSql.checkRows(1)
# tinyint,smallint,int,bigint,tinyint unsigned,smallint unsigned,int unsigned,bigint unsigned
......
tests/system-test/2-query/percentile.py
浏览文件 @ b924ff08
......@@ -23,13 +23,14 @@ from util.sqlset import TDSetSql
class TDTestCase:
def init(self, conn, logSql):
tdLog.debug("start to execute %s" % __file__)
tdSql.init(conn.cursor(), True)
tdSql.init(conn.cursor())
self.rowNum = 10
self.ts = 1537146000000
self.setsql = TDSetSql()
self.ntbname = 'ntb'
self.stbname = 'stb'
self.dbname = 'db'
self.ntbname = f'{self.dbname}.ntb'
self.stbname = f'{self.dbname}.stb'
self.binary_length = 20 # the length of binary for column_dict
self.nchar_length = 20 # the length of nchar for column_dict
self.column_dict = {
......@@ -100,10 +101,9 @@ class TDTestCase:
return intData,floatData
def check_tags(self,tags,param,num,value):
tdSql.query(f'select percentile({tags}, {param}) from {self.stbname}_{num}')
print(tdSql.queryResult)
tdSql.checkEqual(tdSql.queryResult[0][0], value)
def function_check_ntb(self):
tdSql.prepare()
tdSql.execute(f'create database {self.dbname}')
tdSql.execute(self.setsql.set_create_normaltable_sql(self.ntbname,self.column_dict))
intData,floatData = self.insert_data(self.column_dict,self.ntbname,self.rowNum)
for k,v in self.column_dict.items():
......@@ -116,8 +116,9 @@ class TDTestCase:
else:
tdSql.query(f'select percentile({k}, {param}) from {self.ntbname}')
tdSql.checkData(0, 0, np.percentile(floatData, param))
tdSql.execute(f'drop database {self.dbname}')
def function_check_ctb(self):
tdSql.prepare()
tdSql.execute(f'create database {self.dbname}')
tdSql.execute(self.setsql.set_create_stable_sql(self.stbname,self.column_dict,self.tag_dict))
for i in range(self.tbnum):
tdSql.execute(f"create table {self.stbname}_{i} using {self.stbname} tags({self.tag_values[0]})")
......@@ -143,7 +144,7 @@ class TDTestCase:
data_num = tdSql.queryResult[0][0]
tdSql.query(f'select percentile({k},{param}) from {self.stbname}_{i}')
tdSql.checkData(0,0,data_num)
tdSql.execute(f'drop database {self.dbname}')
def run(self):
self.function_check_ntb()
self.function_check_ctb()
......
tests/system-test/2-query/timetruncate.py
浏览文件 @ b924ff08
......@@ -21,9 +21,10 @@ class TDTestCase:
self.db_param_precision = ['ms','us','ns']
self.time_unit = ['1w','1d','1h','1m','1s','1a','1u','1b']
self.error_unit = ['2w','2d','2h','2m','2s','2a','2u','1c','#1']
self.ntbname = 'ntb'
self.stbname = 'stb'
self.ctbname = 'ctb'
self.dbname = 'db'
self.ntbname = f'{self.dbname}.ntb'
self.stbname = f'{self.dbname}.stb'
self.ctbname = f'{self.dbname}.ctb'
def check_ms_timestamp(self,unit,date_time):
if unit.lower() == '1a':
for i in range(len(self.ts_str)):
......@@ -140,9 +141,9 @@ class TDTestCase:
tdSql.error(f'select timetruncate(ts,{unit}) from {self.stbname}')
def function_check_ntb(self):
for precision in self.db_param_precision:
tdSql.execute('drop database if exists db')
tdSql.execute(f'create database db precision "{precision}"')
tdSql.execute('use db')
tdSql.execute(f'drop database if exists {self.dbname}')
tdSql.execute(f'create database {self.dbname} precision "{precision}"')
tdSql.execute(f'use {self.dbname}')
tdSql.execute(f'create table {self.ntbname} (ts timestamp,c0 int)')
for ts in self.ts_str:
tdSql.execute(f'insert into {self.ntbname} values("{ts}",1)')
......@@ -150,9 +151,9 @@ class TDTestCase:
self.data_check(date_time,precision,'ntb')
def function_check_stb(self):
for precision in self.db_param_precision:
tdSql.execute('drop database if exists db')
tdSql.execute(f'create database db precision "{precision}"')
tdSql.execute('use db')
tdSql.execute(f'drop database if exists {self.dbname}')
tdSql.execute(f'create database {self.dbname} precision "{precision}"')
tdSql.execute(f'use {self.dbname}')
tdSql.execute(f'create table {self.stbname} (ts timestamp,c0 int) tags(t0 int)')
tdSql.execute(f'create table {self.ctbname} using {self.stbname} tags(1)')
for ts in self.ts_str:
......
tests/system-test/2-query/timezone.py
浏览文件 @ b924ff08
......@@ -17,10 +17,11 @@ class TDTestCase:
self.setsql = TDSetSql()
self.arithmetic_operators = ['+','-','*','/']
self.arithmetic_values = [0,1,100,15.5]
self.dbname = 'db'
# name of normal table
self.ntbname = 'ntb'
self.ntbname = f'{self.dbname}.ntb'
# name of stable
self.stbname = 'stb'
self.stbname = f'{self.dbname}.stb'
# structure of column
self.column_dict = {
'ts':'timestamp',
......@@ -60,7 +61,6 @@ class TDTestCase:
time_zone_1 = os.popen('ls -l /etc/localtime|awk -F/ \'{print $(NF-1) "/" $NF}\'').read().strip()
time_zone_2 = os.popen('date "+(%Z, %z)"').read().strip()
time_zone = time_zone_1 + " " + time_zone_2
print("expected time zone: " + time_zone)
return time_zone
def tb_type_check(self,tb_type):
......@@ -94,7 +94,7 @@ class TDTestCase:
tdSql.query(f"select * from {tbname} where timezone()='{timezone}'")
self.tb_type_check(tb_type)
def timezone_check_ntb(self,timezone):
tdSql.prepare()
tdSql.execute(f'create database {self.dbname}')
tdSql.execute(self.setsql.set_create_normaltable_sql(self.ntbname,self.column_dict))
for value in self.values_list:
tdSql.execute(
......@@ -102,7 +102,7 @@ class TDTestCase:
self.data_check(timezone,self.ntbname,'normal_table')
tdSql.execute('drop database db')
def timezone_check_stb(self,timezone):
tdSql.prepare()
tdSql.execute(f'create database {self.dbname}')
tdSql.execute(self.setsql.set_create_stable_sql(self.stbname,self.column_dict,self.tag_dict))
for i in range(self.tbnum):
tdSql.execute(f'create table if not exists {self.stbname}_{i} using {self.stbname} tags({self.tag_values[i]})')
......
tests/system-test/2-query/top.py
浏览文件 @ b924ff08
......@@ -23,7 +23,9 @@ class TDTestCase:
tdLog.debug("start to execute %s" % __file__)
tdSql.init(conn.cursor())
self.setsql = TDSetSql()
self.ntbname = 'ntb'
self.dbname = 'db'
self.stbname = f'{self.dbname}.stb'
self.ntbname = f'{self.dbname}.ntb'
self.rowNum = 10
self.tbnum = 20
self.ts = 1537146000000
......@@ -96,52 +98,49 @@ class TDTestCase:
else:
tdSql.checkRows(2*self.tbnum)
def top_check_stb(self):
dbname = tdCom.getLongName(10, "letters")
stbname = tdCom.getLongName(5, "letters")
tag_dict = {
't0':'int'
}
tag_values = [
f'1'
]
tdSql.execute(f"create database if not exists {dbname} vgroups 2")
tdSql.execute(f'use {dbname}')
tdSql.execute(self.setsql.set_create_stable_sql(stbname,self.column_dict,tag_dict))
tdSql.execute(f"create database if not exists {self.dbname} vgroups 2")
tdSql.execute(f'use {self.dbname}')
tdSql.execute(self.setsql.set_create_stable_sql(self.stbname,self.column_dict,tag_dict))
for i in range(self.tbnum):
tdSql.execute(f"create table {stbname}_{i} using {stbname} tags({tag_values[0]})")
self.insert_data(self.column_dict,f'{stbname}_{i}',self.rowNum)
tdSql.query('show tables')
tdSql.execute(f"create table {self.stbname}_{i} using {self.stbname} tags({tag_values[0]})")
self.insert_data(self.column_dict,f'{self.stbname}_{i}',self.rowNum)
tdSql.query(f'show {self.dbname}.tables')
vgroup_list = []
for i in range(len(tdSql.queryResult)):
vgroup_list.append(tdSql.queryResult[i][6])
vgroup_list_set = set(vgroup_list)
for i in vgroup_list_set:
vgroups_num = vgroup_list.count(i)
if vgroups_num >= 2:
tdLog.info(f'This scene with {vgroups_num} vgroups is ok!')
else:
tdLog.exit(
'This scene does not meet the requirements with {vgroups_num} vgroup!\n')
for i in range(self.tbnum):
self.top_check_data(f'{stbname}_{i}','child_table')
self.top_check_data(stbname,'stable')
tdSql.execute(f'drop database {dbname}')
self.top_check_data(f'{self.stbname}_{i}','child_table')
self.top_check_data(self.stbname,'stable')
tdSql.execute(f'drop database {self.dbname}')
def top_check_ntb(self):
tdSql.prepare()
tdSql.execute(f"create database if not exists {self.dbname}")
tdSql.execute(f'use {self.dbname}')
tdSql.execute(self.setsql.set_create_normaltable_sql(self.ntbname,self.column_dict))
self.insert_data(self.column_dict,self.ntbname,self.rowNum)
self.top_check_data(self.ntbname,'normal_table')
tdSql.execute('drop database db')
tdSql.execute(f'drop database {self.dbname}')
def run(self):
self.top_check_ntb()
self.top_check_stb()
def stop(self):
tdSql.close()
tdLog.success("%s successfully executed" % __file__)
......
Markdown is supported
0% 或 .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑！
想要评论请 注册 或