Skip to content

I
Iotdb

apache / Iotdb
11 个月 前同步成功

通知 25
Star 3344
Fork 916
I
Iotdb

体验新版 GitCode，发现更多精彩内容 >>

提交 ea6daf13 编写于 作者: J jt
浏览文件
操作

Merge branch 'master' into dev_merge 

# Conflicts:
#	jdbc/src/test/java/org/apache/iotdb/jdbc/IoTDBQueryResultSetTest.java
#	server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
#	tsfile/src/main/java/org/apache/iotdb/tsfile/write/writer/TsFileIOWriter.java
上级 d1428ba1 d3460734
展开全部 隐藏空白更改
内联 并排
Showing with 989 addition and 700 deletion
client/src/main/java/org/apache/iotdb/client/Client.java
浏览文件 @ ea6daf13
......@@ -33,6 +33,9 @@ import org.apache.iotdb.jdbc.Config;
import org.apache.iotdb.jdbc.IoTDBConnection;
import org.apache.thrift.TException;
/**
* usage: -h 127.0.0.1 -p 6667 -u root -pw root
*/
public class Client extends AbstractClient {
private static CommandLine commandLine;
......
docs/Documentation-CHN/UserGuide/6-JDBC API/2-Status Code.md 0 → 100644
浏览文件 @ ea6daf13
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
# Chapter6: JDBC API
## Status Code
在最新版本中引入了**状态码**这一概念。例如，因为IoTDB需要在写入数据之前首先注册时间序列，一种可能的解决方案是：
```
try {
writeData();
} catch (SQLException e) {
// the most case is that the time series does not exist
if (e.getMessage().contains("exist")) {
//However, using the content of the error message is not so efficient
registerTimeSeries();
//write data once again
writeData();
}
}
```
利用状态码，我们就可以不必写诸如`if (e.getErrorMessage().contains("exist"))`的代码，只需要使用`e.getStatusType().getCode() == TSStatusType.TIME_SERIES_NOT_EXIST_ERROR.getStatusCode()`。
这里是状态码和相对应信息的列表：
|状态码|状态类型|状态信息|
|:---|:---|:---|
|200|SUCCESS_STATUS||
|201|STILL_EXECUTING_STATUS||
|202|INVALID_HANDLE_STATUS||
|301|TIMESERIES_NOT_EXIST_ERROR|时间序列不存在|
|302|UNSUPPORTED_FETCH_METADATA_OPERATION_ERROR|不支持的获取元数据操作|
|303|FETCH_METADATA_ERROR|获取元数据失败|
|304|CHECK_FILE_LEVEL_ERROR|检查文件层级错误|
|400|EXECUTE_STATEMENT_ERROR|执行语句错误|
|401|SQL_PARSE_ERROR|SQL语句分析错误|
|402|GENERATE_TIME_ZONE_ERROR|生成时区错误|
|403|SET_TIME_ZONE_ERROR|设置时区错误|
|500|INTERNAL_SERVER_ERROR|服务器内部错误|
|600|WRONG_LOGIN_PASSWORD_ERROR|用户名或密码错误|
|601|NOT_LOGIN_ERROR|没有登录|
|602|NO_PERMISSION_ERROR|没有操作权限|
|603|UNINITIALIZED_AUTH_ERROR|授权人未初始化|
docs/Documentation/UserGuide/0-Content.md
浏览文件 @ ea6daf13
......@@ -49,10 +49,13 @@
* 2-Reference
# Chapter 6: JDBC API
* 1-JDBC API
# Chapter 7: TsFile
* 2-Status Code
# Chapter 7: Session API
* 1-Session API
# Chapter 8: TsFile
* 1-Installation
* 2-Usage
* 3-Hierarchy
# Chapter 8: System Tools
# Chapter 9: System Tools
* 1-Sync
* 2-Memory Estimation Tool
docs/Documentation/UserGuide/6-JDBC API/2-Status Code.md 0 → 100644
浏览文件 @ ea6daf13
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
# Chapter6: JDBC API
## Status Code
**Status Code** is introduced in the latest version. For example, as IoTDB requires registering the time series first before writing data, a kind of solution is:
```
try {
writeData();
} catch (SQLException e) {
// the most case is that the time series does not exist
if (e.getMessage().contains("exist")) {
//However, using the content of the error message is not so efficient
registerTimeSeries();
//write data once again
writeData();
}
}
```
With Status Code, instead of writing codes like `if (e.getErrorMessage().contains("exist"))`, we can simply use `e.getStatusType().getCode() == TSStatusType.TIME_SERIES_NOT_EXIST_ERROR.getStatusCode()`.
Here is a list of Status Code and related message:
|Status Code|Status Type|Status Message|
|:---|:---|:---|
|200|SUCCESS_STATUS||
|201|STILL_EXECUTING_STATUS||
|202|INVALID_HANDLE_STATUS||
|301|TIMESERIES_NOT_EXIST_ERROR|Timeseries does not exist|
|302|UNSUPPORTED_FETCH_METADATA_OPERATION_ERROR|Unsupported fetch metadata operation|
|303|FETCH_METADATA_ERROR|Failed to fetch metadata|
|304|CHECK_FILE_LEVEL_ERROR|Meet error while checking file level|
|400|EXECUTE_STATEMENT_ERROR|Execute statement error|
|401|SQL_PARSE_ERROR|Meet error while parsing SQL|
|402|GENERATE_TIME_ZONE_ERROR|Meet error while generating time zone|
|403|SET_TIME_ZONE_ERROR|Meet error while setting time zone|
|500|INTERNAL_SERVER_ERROR|Internal server error|
|600|WRONG_LOGIN_PASSWORD_ERROR|Username or password is wrong|
|601|NOT_LOGIN_ERROR|Has not logged in|
|602|NO_PERMISSION_ERROR|No permissions for this operation|
|603|UNINITIALIZED_AUTH_ERROR|Uninitialized authorizer|
docs/Documentation/UserGuide/8-TsFile/3-Hierarchy.md
浏览文件 @ ea6daf13
......@@ -7,9 +7,9 @@
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
......@@ -18,53 +18,63 @@
under the License.
-->
# Chapter 8: TsFile
## TsFile Hierarchy
Here is a brief introduction of the structure of a TsFile file.
## Variable Storage
* **Big Endian**
* For Example, the `int` `0x8` will be stored as `00 00 00 08`, not `08 00 00 00`
* **String with Variable Length**
* The format is `int size` plus `String literal`. Size can be zero.
* Size equals the number of bytes this string will take, and it may not equal to the length of the string.
* For example "sensor_1" will be stored as `00 00 00 08` plus the encoding(ASCII) of "sensor_1".
* Note that for the "Magic String"(file signature) "TsFilev0.8.0", the size(12) and encoding(ASCII)
- **Big Endian**
- For Example, the `int` `0x8` will be stored as `00 00 00 08`, not `08 00 00 00`
- **String with Variable Length**
- The format is `int size` plus `String literal`. Size can be zero.
- Size equals the number of bytes this string will take, and it may not equal to the length of the string.
- For example "sensor_1" will be stored as `00 00 00 08` plus the encoding(ASCII) of "sensor_1".
- Note that for the "Magic String"(file signature) "TsFilev0.8.0", the size(12) and encoding(ASCII)
is fixed so there is no need to put the size before this string literal.
* **Data Type Hardcode**
* 0: BOOLEAN
* 1: INT32 (`int`)
* 2: INT64 (`long`)
* 3: FLOAT
* 4: DOUBLE
* 5: TEXT (`String`)
* **Encoding Type Hardcode**
* 0: PLAIN
* 1: PLAIN_DICTIONARY
* 2: RLE
* 3: DIFF
* 4: TS_2DIFF
* 5: BITMAP
* 6: GORILLA
* 7: REGULAR
* **Compressing Type Hardcode**
* 0: UNCOMPRESSED
* 1: SNAPPY
- **Data Type Hardcode**
- 0: BOOLEAN
- 1: INT32 (`int`)
- 2: INT64 (`long`)
- 3: FLOAT
- 4: DOUBLE
- 5: TEXT (`String`)
- **Encoding Type Hardcode**
- 0: PLAIN
- 1: PLAIN_DICTIONARY
- 2: RLE
- 3: DIFF
- 4: TS_2DIFF
- 5: BITMAP
- 6: GORILLA
- 7: REGULAR
- **Compressing Type Hardcode**
- 0: UNCOMPRESSED
- 1: SNAPPY
- 2: GZIP
- 3: LZO
- 4: SDT
- 5: PAA
- 6: PLA
- **TsDigest Statistics Type Hardcode**
- 0: min_value
- 1: max_value
- 2: first_value
- 3: last_value
- 4: sum_value
## TsFile Overview
Here is a graph about the TsFile structure.
![TsFile Breakdown](https://user-images.githubusercontent.com/40447846/61616997-6fad1300-ac9c-11e9-9c17-46785ebfbc88.png)
## Magic String
There is a 12 bytes magic string:
`TsFilev0.8.0`
......@@ -87,6 +97,7 @@ The `ChunkGroup` has an array of `Chunk`, a following byte `0x00` as the marker,
A `Chunk` represents a *sensor*. There is a byte `0x01` as the marker, following a `ChunkHeader` and an array of `Page`.
###### ChunkHeader
<center>
<table style="text-align:center">
<tr><th>Member Description</th><th>Member Type</td></tr>
......@@ -136,6 +147,7 @@ PageHeader Structure
## Metadata
### TsDeviceMetaData
The first part of metadata is `TsDeviceMetaData`
<center>
......@@ -148,6 +160,7 @@ The first part of metadata is `TsDeviceMetaData`
</center>
Then there is an array of `ChunkGroupMetaData` after `TsDeviceMetaData`
### ChunkGroupMetaData
<center>
......@@ -179,14 +192,57 @@ Then there is an array of `ChunkMetadata` for each `ChunkGroupMetadata`
</table>
</center>
###### TsDigest
###### TsDigest (updated on 2019/8/27)
Right now there are five statistics: `min_value, max_value, first_value, last_value, sum_value`.
In v0.8.0, the storage format of statistics is a name-value pair. That is, `Map<String, ByteBuffer> statistics`. The name is a string (remember the length is before the literal). But for the value, there is also an integer byteLength acting as the self description length of the following value because the value may be of various type. For example, if the `min_value` is an integer 0, then it will be stored as [9 "min_value" 4 0] in the TsFile.
The figure below shows an example of `TsDigest.deserializeFrom(buffer)`. In v0.8.0, we will get
There are five statistics: `min, last, sum, first, max`
```
Map<String, ByteBuffer> statistics = {
"min_value" -> ByteBuffer of int value 0,
"last" -> ByteBuffer of int value 19,
"sum" -> ByteBuffer of double value 1093347116,
"first" -> ByteBuffer of int value 0,
"max_value" -> ByteBuffer of int value 99
}
```
The storage format is a name-value pair. The name is a string (remember the length is before the literal).
![TsDigest ByteBuffer Breakdown comparison](https://user-images.githubusercontent.com/33376433/63765352-664a4280-c8fb-11e9-869e-859edf6d00bb.png)
But for the value, there is also a size integer before the data even if it is not string. For example, if the `min` is 3, then it will be
stored as 3 "min" 4 3 in the TsFile.
In v0.9.0, the storage format is changed to an array for space and time efficiency. That is, `ByteBuffer[] statistics`. Each position of the array has a fixed association with a specific type of statistic, following the order defined in StatisticType:
```
enum StatisticType {
min_value, max_value, first_value, last_value, sum_value
}
```
Therefore, in the above example, we will get
```
ByteBuffer[] statistics = [
ByteBuffer of int value 0, // associated with "min_value"
ByteBuffer of int value 99, // associated with "max_value"
ByteBuffer of int value 0, // associated with "first_value"
ByteBuffer of int value 19, // associated with "last_value"
ByteBuffer of double value 1093347116 // associated with "sum_value"
]
```
As another example in v0.9.0, when deserializing a TsDigest from buffer [3, 0,4,0, 1,4,99, 3,4,19], we get
```
ByteBuffer[] statistics = [
ByteBuffer of int value 0, // associated with "min_value"
ByteBuffer of int value 99, // associated with "max_value"
null, // associated with "first_value"
ByteBuffer of int value 19, // associated with "last_value"
null // associated with "sum_value"
]
```
#### File Metadata
......@@ -207,6 +263,7 @@ After the array of `ChunkGroupMetadata`, here is the last part of the metadata.
</center>
##### DeviceIndexMetadata
<center>
<table style="text-align:center">
<tr><th>Member Description</th><th>Member Type</td></tr>
......@@ -219,6 +276,7 @@ After the array of `ChunkGroupMetadata`, here is the last part of the metadata.
</center>
##### MeasurementSchema
<center>
<table style="text-align:center">
<tr><th>Member Description</th><th>Member Type</td></tr>
......
jdbc/src/main/java/org/apache/iotdb/jdbc/IoTDBConnection.java
浏览文件 @ ea6daf13
......@@ -424,7 +424,7 @@ public class IoTDBConnection implements Connection {
} catch (IoTDBRPCException e) {
// failed to connect, disconnect from the server
transport.close();
throw new IoTDBSQLException(e);
throw new IoTDBSQLException(e.getMessage());
}
if (protocolVersion.getValue() != openResp.getServerProtocolVersion().getValue()) {
throw new TException(String
......@@ -485,7 +485,7 @@ public class IoTDBConnection implements Connection {
try {
RpcUtils.verifySuccess(resp.getStatus());
} catch (IoTDBRPCException e) {
throw new IoTDBSQLException(e);
throw new IoTDBSQLException(e.getMessage());
}
return resp.getTimeZone();
}
......@@ -496,7 +496,7 @@ public class IoTDBConnection implements Connection {
try {
RpcUtils.verifySuccess(resp.getStatus());
} catch (IoTDBRPCException e) {
throw new IoTDBSQLException(e);
throw new IoTDBSQLException(e.getMessage());
}
this.zoneId = ZoneId.of(zoneId);
}
......
jdbc/src/main/java/org/apache/iotdb/jdbc/IoTDBDatabaseMetadata.java
浏览文件 @ ea6daf13
......@@ -87,7 +87,7 @@ public class IoTDBDatabaseMetadata implements DatabaseMetaData {
try {
RpcUtils.verifySuccess(resp.getStatus());
} catch (IoTDBRPCException e) {
throw new IoTDBSQLException(e);
throw new IoTDBSQLException(e.getMessage());
}
return new IoTDBMetadataResultSet(resp.getColumnsList(), null, null);
} catch (TException e) {
......@@ -101,7 +101,7 @@ public class IoTDBDatabaseMetadata implements DatabaseMetaData {
try {
RpcUtils.verifySuccess(resp.getStatus());
} catch (IoTDBRPCException e) {
throw new IoTDBSQLException(e);
throw new IoTDBSQLException(e.getMessage());
}
return new IoTDBMetadataResultSet(resp.getColumnsList(), null, null);
} catch (TException e) {
......@@ -114,7 +114,7 @@ public class IoTDBDatabaseMetadata implements DatabaseMetaData {
try {
RpcUtils.verifySuccess(resp.getStatus());
} catch (IoTDBRPCException e) {
throw new IoTDBSQLException(e);
throw new IoTDBSQLException(e.getMessage());
}
Set<String> showStorageGroup = resp.getShowStorageGroups();
return new IoTDBMetadataResultSet(null, showStorageGroup, null);
......@@ -129,7 +129,7 @@ public class IoTDBDatabaseMetadata implements DatabaseMetaData {
try {
RpcUtils.verifySuccess(resp.getStatus());
} catch (IoTDBRPCException e) {
throw new IoTDBSQLException(e);
throw new IoTDBSQLException(e.getMessage());
}
List<List<String>> showTimeseriesList = resp.getShowTimeseriesList();
return new IoTDBMetadataResultSet(null, null, showTimeseriesList);
......@@ -1268,7 +1268,7 @@ public class IoTDBDatabaseMetadata implements DatabaseMetaData {
try {
RpcUtils.verifySuccess(resp.getStatus());
} catch (IoTDBRPCException e) {
throw new IoTDBSQLException(e);
throw new IoTDBSQLException(e.getMessage());
}
return resp.getMetadataInJson();
}
......
jdbc/src/main/java/org/apache/iotdb/jdbc/IoTDBPreparedInsertionStatement.java
浏览文件 @ ea6daf13
......@@ -22,11 +22,12 @@ package org.apache.iotdb.jdbc;
import java.sql.SQLException;
import java.time.ZoneId;
import java.util.List;
import org.apache.iotdb.rpc.TSStatusType;
import org.apache.iotdb.service.rpc.thrift.TSExecuteStatementResp;
import org.apache.iotdb.service.rpc.thrift.TSIService.Iface;
import org.apache.iotdb.service.rpc.thrift.TSInsertionReq;
import org.apache.iotdb.service.rpc.thrift.TS_SessionHandle;
import org.apache.iotdb.service.rpc.thrift.TS_StatusCode;
import org.apache.thrift.TException;
public class IoTDBPreparedInsertionStatement extends IoTDBPreparedStatement {
......@@ -49,7 +50,7 @@ public class IoTDBPreparedInsertionStatement extends IoTDBPreparedStatement {
req.unsetMeasurements();
req.unsetTimestamp();
req.unsetValues();
return resp.getStatus().getStatusCode() == TS_StatusCode.SUCCESS_STATUS;
return resp.getStatus().getStatusType().getCode() == TSStatusType.SUCCESS_STATUS.getStatusCode();
} catch (TException e) {
throw new SQLException(e);
}
......
jdbc/src/main/java/org/apache/iotdb/jdbc/IoTDBQueryResultSet.java
浏览文件 @ ea6daf13
......@@ -713,7 +713,7 @@ public class IoTDBQueryResultSet implements ResultSet {
try {
RpcUtils.verifySuccess(resp.getStatus());
} catch (IoTDBRPCException e) {
throw new IoTDBSQLException(e);
throw new IoTDBSQLException(e.getMessage());
}
if (!resp.hasResultSet) {
emptyResultSet = true;
......
jdbc/src/main/java/org/apache/iotdb/jdbc/IoTDBStatement.java
浏览文件 @ ea6daf13
......@@ -30,20 +30,11 @@ import java.time.ZoneId;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.atomic.AtomicLong;
import org.apache.iotdb.rpc.IoTDBRPCException;
import org.apache.iotdb.rpc.RpcUtils;
import org.apache.iotdb.service.rpc.thrift.TSCancelOperationReq;
import org.apache.iotdb.service.rpc.thrift.TSCancelOperationResp;
import org.apache.iotdb.service.rpc.thrift.TSCloseOperationReq;
import org.apache.iotdb.service.rpc.thrift.TSCloseOperationResp;
import org.apache.iotdb.service.rpc.thrift.TSExecuteBatchStatementReq;
import org.apache.iotdb.service.rpc.thrift.TSExecuteBatchStatementResp;
import org.apache.iotdb.service.rpc.thrift.TSExecuteStatementReq;
import org.apache.iotdb.service.rpc.thrift.TSExecuteStatementResp;
import org.apache.iotdb.service.rpc.thrift.TSIService;
import org.apache.iotdb.service.rpc.thrift.TSOperationHandle;
import org.apache.iotdb.service.rpc.thrift.TS_SessionHandle;
import org.apache.iotdb.service.rpc.thrift.TS_StatusCode;
import org.apache.iotdb.rpc.TSStatusType;
import org.apache.iotdb.service.rpc.thrift.*;
import org.apache.thrift.TException;
public class IoTDBStatement implements Statement {
......@@ -51,6 +42,7 @@ public class IoTDBStatement implements Statement {
private static final String SHOW_TIMESERIES_COMMAND_LOWERCASE = "show timeseries";
private static final String SHOW_STORAGE_GROUP_COMMAND_LOWERCASE = "show storage group";
private static final String METHOD_NOT_SUPPORTED_STRING = "Method not supported";
ZoneId zoneId;
private ResultSet resultSet = null;
private IoTDBConnection connection;
......@@ -251,7 +243,7 @@ public class IoTDBStatement implements Statement {
try {
RpcUtils.verifySuccess(execResp.getStatus());
} catch (IoTDBRPCException e) {
throw new IoTDBSQLException(e);
throw new IoTDBSQLException(e.getMessage());
}
if (execResp.getOperationHandle().hasResultSet) {
IoTDBQueryResultSet resSet = new IoTDBQueryResultSet(this,
......@@ -294,7 +286,7 @@ public class IoTDBStatement implements Statement {
TSExecuteBatchStatementReq execReq = new TSExecuteBatchStatementReq(sessionHandle,
batchSQLList);
TSExecuteBatchStatementResp execResp = client.executeBatchStatement(execReq);
if (execResp.getStatus().statusCode == TS_StatusCode.SUCCESS_STATUS) {
if (execResp.getStatus().getStatusType().getCode() == TSStatusType.SUCCESS_STATUS.getStatusCode()) {
if (execResp.getResult() == null) {
return new int[0];
} else {
......@@ -309,7 +301,7 @@ public class IoTDBStatement implements Statement {
} else {
BatchUpdateException exception;
if (execResp.getResult() == null) {
exception = new BatchUpdateException(execResp.getStatus().errorMessage, new int[0]);
exception = new BatchUpdateException(execResp.getStatus().getStatusType().getMessage(), new int[0]);
} else {
List<Integer> result = execResp.getResult();
int len = result.size();
......@@ -317,7 +309,7 @@ public class IoTDBStatement implements Statement {
for (int i = 0; i < len; i++) {
updateArray[i] = result.get(i);
}
exception = new BatchUpdateException(execResp.getStatus().errorMessage, updateArray);
exception = new BatchUpdateException(execResp.getStatus().getStatusType().getMessage(), updateArray);
}
throw exception;
}
......@@ -355,7 +347,7 @@ public class IoTDBStatement implements Statement {
try {
RpcUtils.verifySuccess(execResp.getStatus());
} catch (IoTDBRPCException e) {
throw new IoTDBSQLException(e);
throw new IoTDBSQLException(e.getMessage());
}
IoTDBQueryResultSet resSet = new IoTDBQueryResultSet(this, execResp.getColumns(), client,
operationHandle, sql, execResp.getOperationType(), execResp.getDataTypeList(),
......@@ -412,7 +404,7 @@ public class IoTDBStatement implements Statement {
try {
RpcUtils.verifySuccess(execResp.getStatus());
} catch (IoTDBRPCException e) {
throw new IoTDBSQLException(e);
throw new IoTDBSQLException(e.getMessage());
}
return 0;
}
......
jdbc/src/main/java/org/apache/iotdb/jdbc/Utils.java
浏览文件 @ ea6daf13
......@@ -18,7 +18,6 @@
*/
package org.apache.iotdb.jdbc;
import java.nio.ByteBuffer;
import java.util.ArrayList;
import java.util.List;
import java.util.Properties;
......@@ -32,8 +31,6 @@ import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
import org.apache.iotdb.tsfile.read.common.Field;
import org.apache.iotdb.tsfile.read.common.RowRecord;
import org.apache.iotdb.tsfile.utils.Binary;
import org.apache.iotdb.tsfile.utils.BytesUtils;
import org.apache.iotdb.tsfile.write.record.RowBatch;
/**
* Utils to convert between thrift format and TsFile format.
......
jdbc/src/test/java/org/apache/iotdb/jdbc/BatchTest.java
浏览文件 @ ea6daf13
......@@ -29,12 +29,9 @@ import java.sql.Statement;
import java.time.ZoneId;
import java.util.ArrayList;
import java.util.List;
import org.apache.iotdb.service.rpc.thrift.TSExecuteBatchStatementReq;
import org.apache.iotdb.service.rpc.thrift.TSExecuteBatchStatementResp;
import org.apache.iotdb.service.rpc.thrift.TSIService;
import org.apache.iotdb.service.rpc.thrift.TS_SessionHandle;
import org.apache.iotdb.service.rpc.thrift.TS_Status;
import org.apache.iotdb.service.rpc.thrift.TS_StatusCode;
import org.apache.iotdb.rpc.TSStatusType;
import org.apache.iotdb.service.rpc.thrift.*;
import org.apache.thrift.TException;
import org.junit.After;
import org.junit.Before;
......@@ -50,8 +47,10 @@ public class BatchTest {
private TSIService.Iface client;
@Mock
private TS_SessionHandle sessHandle;
private TS_Status Status_SUCCESS = new TS_Status(TS_StatusCode.SUCCESS_STATUS);
private TS_Status Status_ERROR = new TS_Status(TS_StatusCode.ERROR_STATUS);
private TS_StatusType successStatus = new TS_StatusType(TSStatusType.SUCCESS_STATUS.getStatusCode(), "");
private TS_StatusType errorStatus = new TS_StatusType(TSStatusType.INTERNAL_SERVER_ERROR.getStatusCode(), "");
private TS_Status Status_SUCCESS = new TS_Status(successStatus);
private TS_Status Status_ERROR = new TS_Status(errorStatus);
private TSExecuteBatchStatementResp resp;
private ZoneId zoneID = ZoneId.systemDefault();
......
jdbc/src/test/java/org/apache/iotdb/jdbc/IoTDBConnectionTest.java
浏览文件 @ ea6daf13
......@@ -24,13 +24,9 @@ import static org.mockito.Mockito.when;
import java.util.ArrayList;
import java.util.List;
import org.apache.iotdb.service.rpc.thrift.ServerProperties;
import org.apache.iotdb.service.rpc.thrift.TSGetTimeZoneResp;
import org.apache.iotdb.service.rpc.thrift.TSIService;
import org.apache.iotdb.service.rpc.thrift.TSSetTimeZoneReq;
import org.apache.iotdb.service.rpc.thrift.TSSetTimeZoneResp;
import org.apache.iotdb.service.rpc.thrift.TS_Status;
import org.apache.iotdb.service.rpc.thrift.TS_StatusCode;
import org.apache.iotdb.rpc.TSStatusType;
import org.apache.iotdb.service.rpc.thrift.*;
import org.apache.thrift.TException;
import org.junit.After;
import org.junit.Before;
......@@ -44,7 +40,8 @@ public class IoTDBConnectionTest {
private TSIService.Iface client;
private IoTDBConnection connection = new IoTDBConnection();
private TS_Status Status_SUCCESS = new TS_Status(TS_StatusCode.SUCCESS_STATUS);
private TS_StatusType successStatus = new TS_StatusType(TSStatusType.SUCCESS_STATUS.getStatusCode(), "");
private TS_Status Status_SUCCESS = new TS_Status(successStatus);
@Before
public void setUp() throws Exception {
......
jdbc/src/test/java/org/apache/iotdb/jdbc/IoTDBDatabaseMetadataTest.java
浏览文件 @ ea6daf13
......@@ -30,11 +30,9 @@ import java.util.ArrayList;
import java.util.HashSet;
import java.util.List;
import java.util.Set;
import org.apache.iotdb.service.rpc.thrift.TSFetchMetadataReq;
import org.apache.iotdb.service.rpc.thrift.TSFetchMetadataResp;
import org.apache.iotdb.service.rpc.thrift.TSIService;
import org.apache.iotdb.service.rpc.thrift.TS_Status;
import org.apache.iotdb.service.rpc.thrift.TS_StatusCode;
import org.apache.iotdb.rpc.TSStatusType;
import org.apache.iotdb.service.rpc.thrift.*;
import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;
......@@ -66,7 +64,8 @@ public class IoTDBDatabaseMetadataTest {
@Mock
private TSFetchMetadataResp fetchMetadataResp;
private TS_Status Status_SUCCESS = new TS_Status(TS_StatusCode.SUCCESS_STATUS);
private TS_StatusType successStatus = new TS_StatusType(TSStatusType.SUCCESS_STATUS.getStatusCode(), "");
private TS_Status Status_SUCCESS = new TS_Status(successStatus);
private DatabaseMetaData databaseMetaData;
......
jdbc/src/test/java/org/apache/iotdb/jdbc/IoTDBPreparedStatementTest.java
浏览文件 @ ea6daf13
......@@ -26,14 +26,10 @@ import static org.mockito.Mockito.when;
import java.sql.SQLException;
import java.sql.Timestamp;
import java.time.ZoneId;
import org.apache.iotdb.service.rpc.thrift.TSExecuteStatementReq;
import org.apache.iotdb.service.rpc.thrift.TSExecuteStatementResp;
import org.apache.iotdb.service.rpc.thrift.TSGetOperationStatusResp;
import org.apache.iotdb.rpc.TSStatusType;
import org.apache.iotdb.service.rpc.thrift.*;
import org.apache.iotdb.service.rpc.thrift.TSIService.Iface;
import org.apache.iotdb.service.rpc.thrift.TSOperationHandle;
import org.apache.iotdb.service.rpc.thrift.TS_SessionHandle;
import org.apache.iotdb.service.rpc.thrift.TS_Status;
import org.apache.iotdb.service.rpc.thrift.TS_StatusCode;
import org.junit.Before;
import org.junit.Test;
import org.mockito.ArgumentCaptor;
......@@ -53,7 +49,8 @@ public class IoTDBPreparedStatementTest {
private Iface client;
@Mock
private TS_SessionHandle sessHandle;
private TS_Status Status_SUCCESS = new TS_Status(TS_StatusCode.SUCCESS_STATUS);
private TS_StatusType successStatus = new TS_StatusType(TSStatusType.SUCCESS_STATUS.getStatusCode(), "");
private TS_Status Status_SUCCESS = new TS_Status(successStatus);
@Mock
private TSOperationHandle tOperationHandle;
......
jdbc/src/test/java/org/apache/iotdb/jdbc/IoTDBQueryResultSetTest.java
浏览文件 @ ea6daf13
......@@ -32,22 +32,10 @@ import java.sql.Types;
import java.time.ZoneId;
import java.util.ArrayList;
import java.util.List;
import org.apache.iotdb.service.rpc.thrift.TSCloseOperationReq;
import org.apache.iotdb.service.rpc.thrift.TSCloseOperationResp;
import org.apache.iotdb.service.rpc.thrift.TSDataValue;
import org.apache.iotdb.service.rpc.thrift.TSExecuteStatementReq;
import org.apache.iotdb.service.rpc.thrift.TSExecuteStatementResp;
import org.apache.iotdb.service.rpc.thrift.TSFetchMetadataReq;
import org.apache.iotdb.service.rpc.thrift.TSFetchMetadataResp;
import org.apache.iotdb.service.rpc.thrift.TSFetchResultsReq;
import org.apache.iotdb.service.rpc.thrift.TSFetchResultsResp;
import org.apache.iotdb.service.rpc.thrift.TSIService;
import org.apache.iotdb.service.rpc.thrift.TSOperationHandle;
import org.apache.iotdb.service.rpc.thrift.TSQueryDataSet;
import org.apache.iotdb.service.rpc.thrift.TSRowRecord;
import org.apache.iotdb.service.rpc.thrift.TS_SessionHandle;
import org.apache.iotdb.service.rpc.thrift.TS_Status;
import org.apache.iotdb.service.rpc.thrift.TS_StatusCode;
import org.apache.iotdb.rpc.TSStatusType;
import org.apache.iotdb.service.rpc.thrift.*;
import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
import org.junit.Assert;
import org.junit.Before;
......@@ -120,7 +108,8 @@ public class IoTDBQueryResultSetTest {
@Mock
private TSFetchResultsResp fetchResultsResp;
private TS_Status Status_SUCCESS = new TS_Status(TS_StatusCode.SUCCESS_STATUS);
private TS_StatusType successStatus = new TS_StatusType(TSStatusType.SUCCESS_STATUS.getStatusCode(), "");
private TS_Status Status_SUCCESS = new TS_Status(successStatus);
private ZoneId zoneID = ZoneId.systemDefault();
@Before
......
jdbc/src/test/java/org/apache/iotdb/jdbc/IoTDBStatementTest.java
浏览文件 @ ea6daf13
......@@ -31,12 +31,10 @@ import java.util.ArrayList;
import java.util.HashSet;
import java.util.List;
import java.util.Set;
import org.apache.iotdb.service.rpc.thrift.TSFetchMetadataReq;
import org.apache.iotdb.service.rpc.thrift.TSFetchMetadataResp;
import org.apache.iotdb.rpc.TSStatusType;
import org.apache.iotdb.service.rpc.thrift.*;
import org.apache.iotdb.service.rpc.thrift.TSIService.Iface;
import org.apache.iotdb.service.rpc.thrift.TS_SessionHandle;
import org.apache.iotdb.service.rpc.thrift.TS_Status;
import org.apache.iotdb.service.rpc.thrift.TS_StatusCode;
import org.apache.thrift.TException;
import org.junit.After;
import org.junit.Assert;
......@@ -59,7 +57,8 @@ public class IoTDBStatementTest {
@Mock
private TSFetchMetadataResp fetchMetadataResp;
private TS_Status Status_SUCCESS = new TS_Status(TS_StatusCode.SUCCESS_STATUS);
private TS_StatusType successStatus = new TS_StatusType(TSStatusType.SUCCESS_STATUS.getStatusCode(), "");
private TS_Status Status_SUCCESS = new TS_Status(successStatus);
private ZoneId zoneID = ZoneId.systemDefault();
@Before
......
jdbc/src/test/java/org/apache/iotdb/jdbc/UtilsTest.java
浏览文件 @ ea6daf13
......@@ -27,11 +27,14 @@ import java.util.ArrayList;
import java.util.List;
import java.util.Properties;
import org.apache.iotdb.rpc.RpcUtils;
import org.apache.iotdb.rpc.TSStatusType;
import org.apache.iotdb.service.rpc.thrift.TSDataValue;
import org.apache.iotdb.service.rpc.thrift.TSQueryDataSet;
import org.apache.iotdb.service.rpc.thrift.TSRowRecord;
import org.apache.iotdb.service.rpc.thrift.TS_Status;
import org.apache.iotdb.service.rpc.thrift.TS_StatusCode;
import org.apache.iotdb.rpc.RpcUtils;
import org.apache.iotdb.service.rpc.thrift.*;
import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
import org.apache.iotdb.tsfile.read.common.Field;
import org.apache.iotdb.tsfile.read.common.RowRecord;
......@@ -70,13 +73,15 @@ public class UtilsTest {
@Test
public void testVerifySuccess() {
try {
RpcUtils.verifySuccess(new TS_Status(TS_StatusCode.SUCCESS_STATUS));
TS_StatusType successStatus = new TS_StatusType(TSStatusType.SUCCESS_STATUS.getStatusCode(), "");
RpcUtils.verifySuccess(new TS_Status(successStatus));
} catch (Exception e) {
fail();
}
try {
RpcUtils.verifySuccess(new TS_Status(TS_StatusCode.ERROR_STATUS));
TS_StatusType errorStatus = new TS_StatusType(TSStatusType.INTERNAL_SERVER_ERROR.getStatusCode(), "");
RpcUtils.verifySuccess(new TS_Status(errorStatus));
} catch (Exception e) {
return;
}
......
server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
浏览文件 @ ea6daf13
......@@ -67,9 +67,9 @@ import org.apache.iotdb.db.qp.physical.crud.DeletePlan;
import org.apache.iotdb.db.qp.physical.crud.InsertPlan;
import org.apache.iotdb.db.query.context.QueryContext;
import org.apache.iotdb.db.query.control.JobFileManager;
import org.apache.iotdb.rpc.TSStatusType;
import org.apache.iotdb.db.utils.CopyOnReadLinkedList;
import org.apache.iotdb.db.writelog.recover.TsFileRecoverPerformer;
import org.apache.iotdb.service.rpc.thrift.TS_StatusCode;
import org.apache.iotdb.tsfile.file.metadata.ChunkMetaData;
import org.apache.iotdb.tsfile.file.metadata.enums.CompressionType;
import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
......@@ -361,7 +361,7 @@ public class StorageGroupProcessor {
List<Integer> unsequenceIndexes = new ArrayList<>();
for (int i = 0; i < batchInsertPlan.getRowCount(); i++) {
results[i] = TS_StatusCode.SUCCESS_STATUS.getValue();
results[i] = TSStatusType.SUCCESS_STATUS.getStatusCode();
if (batchInsertPlan.getTimes()[i] > latestFlushedTimeForEachDevice
.get(batchInsertPlan.getDeviceId())) {
sequenceIndexes.add(i);
......@@ -389,7 +389,7 @@ public class StorageGroupProcessor {
TsFileProcessor tsFileProcessor = getOrCreateTsFileProcessor(sequence);
if (tsFileProcessor == null) {
for (int index : indexes) {
results[index] = TS_StatusCode.ERROR_STATUS.getValue();
results[index] = TSStatusType.INTERNAL_SERVER_ERROR.getStatusCode();
}
return;
}
......
server/src/main/java/org/apache/iotdb/db/engine/storagegroup/TsFileProcessor.java
浏览文件 @ ea6daf13
......@@ -48,10 +48,10 @@ import org.apache.iotdb.db.qp.physical.crud.BatchInsertPlan;
import org.apache.iotdb.db.qp.physical.crud.InsertPlan;
import org.apache.iotdb.db.query.context.QueryContext;
import org.apache.iotdb.db.rescon.MemTablePool;
import org.apache.iotdb.rpc.TSStatusType;
import org.apache.iotdb.db.utils.QueryUtils;
import org.apache.iotdb.db.writelog.manager.MultiFileLogNodeManager;
import org.apache.iotdb.db.writelog.node.WriteLogNode;
import org.apache.iotdb.service.rpc.thrift.TS_StatusCode;
import org.apache.iotdb.tsfile.file.metadata.ChunkMetaData;
import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
import org.apache.iotdb.tsfile.utils.Pair;
......@@ -173,7 +173,7 @@ public class TsFileProcessor {
} catch (IOException e) {
logger.error("write WAL failed", e);
for (int index: indexes) {
results[index] = TS_StatusCode.ERROR_STATUS.getValue();
results[index] = TSStatusType.INTERNAL_SERVER_ERROR.getStatusCode();
}
return false;
}
......
server/src/main/java/org/apache/iotdb/db/query/reader/resourceRelated/UnseqResourceMergeReader.java
浏览文件 @ ea6daf13
......@@ -16,9 +16,11 @@
* specific language governing permissions and limitations
* under the License.
*/
package org.apache.iotdb.db.query.reader.resourceRelated;
import java.io.IOException;
import java.nio.ByteBuffer;
import java.util.List;
import org.apache.iotdb.db.engine.cache.DeviceMetaDataCache;
import org.apache.iotdb.db.engine.modification.Modification;
......@@ -29,8 +31,8 @@ import org.apache.iotdb.db.query.reader.chunkRelated.DiskChunkReader;
import org.apache.iotdb.db.query.reader.chunkRelated.MemChunkReader;
import org.apache.iotdb.db.query.reader.universal.PriorityMergeReader;
import org.apache.iotdb.db.utils.QueryUtils;
import org.apache.iotdb.tsfile.common.constant.StatisticConstant;
import org.apache.iotdb.tsfile.file.metadata.ChunkMetaData;
import org.apache.iotdb.tsfile.file.metadata.TsDigest.StatisticType;
import org.apache.iotdb.tsfile.read.TsFileSequenceReader;
import org.apache.iotdb.tsfile.read.common.Chunk;
import org.apache.iotdb.tsfile.read.common.Path;
......@@ -96,11 +98,16 @@ public class UnseqResourceMergeReader extends PriorityMergeReader {
for (ChunkMetaData chunkMetaData : metaDataList) {
if (filter != null) {
ByteBuffer minValue = null;
ByteBuffer maxValue = null;
ByteBuffer[] statistics = chunkMetaData.getDigest().getStatistics();
if (statistics != null) {
minValue = statistics[StatisticType.min_value.ordinal()]; // note still CAN be null
maxValue = statistics[StatisticType.max_value.ordinal()]; // note still CAN be null
}
DigestForFilter digest = new DigestForFilter(chunkMetaData.getStartTime(),
chunkMetaData.getEndTime(),
chunkMetaData.getDigest().getStatistics().get(StatisticConstant.MIN_VALUE),
chunkMetaData.getDigest().getStatistics().get(StatisticConstant.MAX_VALUE),
chunkMetaData.getTsDataType());
chunkMetaData.getEndTime(), minValue, maxValue, chunkMetaData.getTsDataType());
if (!filter.satisfy(digest)) {
continue;
}
......
server/src/test/java/org/apache/iotdb/db/engine/merge/MaxSeriesMergeFileSelectorTest.java
浏览文件 @ ea6daf13
......@@ -97,7 +97,7 @@ public class MaxSeriesMergeFileSelectorTest extends MergeTest{
List<TsFileResource> unseqSelected = result[1];
assertEquals(seqResources.subList(0, 1), seqSelected);
assertEquals(unseqResources.subList(0, 1), unseqSelected);
assertEquals(12, mergeFileSelector.getConcurrentMergeNum());
assertEquals(34, mergeFileSelector.getConcurrentMergeNum());
resource.clear();
}
}
server/src/test/java/org/apache/iotdb/db/integration/IoTDBAggregationIT.java
浏览文件 @ ea6daf13
......@@ -471,7 +471,7 @@ public class IoTDBAggregationIT {
resultSet.next();
fail();
} catch (Exception e) {
Assert.assertEquals("org.apache.iotdb.rpc.IoTDBRPCException: Unsupported data type in aggregation MEAN : TEXT", e.getMessage());
Assert.assertEquals("Internal server error: Unsupported data type in aggregation MEAN : TEXT", e.getMessage());
}
hasResultSet = statement.execute("select sum(s3)" +
......@@ -481,7 +481,7 @@ public class IoTDBAggregationIT {
resultSet.next();
fail();
} catch (Exception e) {
Assert.assertEquals("org.apache.iotdb.rpc.IoTDBRPCException: Unsupported data type in aggregation SUM : TEXT", e.getMessage());
Assert.assertEquals("Internal server error: Unsupported data type in aggregation SUM : TEXT", e.getMessage());
}
hasResultSet = statement.execute("select mean(s4)" +
......@@ -492,7 +492,7 @@ public class IoTDBAggregationIT {
resultSet.next();
fail();
} catch (Exception e) {
Assert.assertEquals("org.apache.iotdb.rpc.IoTDBRPCException: Unsupported data type in aggregation MEAN : BOOLEAN", e.getMessage());
Assert.assertEquals("Internal server error: Unsupported data type in aggregation MEAN : BOOLEAN", e.getMessage());
}
hasResultSet = statement.execute("select sum(s4)" +
......@@ -502,7 +502,7 @@ public class IoTDBAggregationIT {
resultSet.next();
fail();
} catch (Exception e) {
Assert.assertEquals("org.apache.iotdb.rpc.IoTDBRPCException: Unsupported data type in aggregation SUM : BOOLEAN", e.getMessage());
Assert.assertEquals("Internal server error: Unsupported data type in aggregation SUM : BOOLEAN", e.getMessage());
}
} catch (Exception e) {
e.printStackTrace();
......
service-rpc/src/main/java/org/apache/iotdb/rpc/RpcUtils.java
浏览文件 @ ea6daf13
......@@ -21,7 +21,6 @@ package org.apache.iotdb.rpc;
import java.lang.reflect.Proxy;
import org.apache.iotdb.service.rpc.thrift.TSIService;
import org.apache.iotdb.service.rpc.thrift.TS_Status;
import org.apache.iotdb.service.rpc.thrift.TS_StatusCode;
public class RpcUtils {
......@@ -36,8 +35,8 @@ public class RpcUtils {
* @param status -status
*/
public static void verifySuccess(TS_Status status) throws IoTDBRPCException {
if (status.getStatusCode() != TS_StatusCode.SUCCESS_STATUS) {
throw new IoTDBRPCException(status.errorMessage);
if (status.getStatusType().getCode() != TSStatusType.SUCCESS_STATUS.getStatusCode()) {
throw new IoTDBRPCException(status.getStatusType().getMessage());
}
}
......
service-rpc/src/main/java/org/apache/iotdb/rpc/TSStatusType.java 0 → 100644
浏览文件 @ ea6daf13
/**
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
package org.apache.iotdb.rpc;
public enum TSStatusType {
SUCCESS_STATUS(200, ""),
STILL_EXECUTING_STATUS(201, ""),
INVALID_HANDLE_STATUS(202, ""),
TIMESERIES_NOT_EXIST_ERROR(301, "Timeseries does not exist"),
UNSUPPORTED_FETCH_METADATA_OPERATION_ERROR(302, "Unsupported fetch metadata operation"),
FETCH_METADATA_ERROR(303, "Failed to fetch metadata"),
CHECK_FILE_LEVEL_ERROR(304, "Meet error while checking file level"),
EXECUTE_STATEMENT_ERROR(400, "Execute statement error"),
SQL_PARSE_ERROR(401, "Meet error while parsing SQL"),
GENERATE_TIME_ZONE_ERROR(402, "Meet error while generating time zone"),
SET_TIME_ZONE_ERROR(403, "Meet error while setting time zone"),
INTERNAL_SERVER_ERROR(500, "Internal server error"),
WRONG_LOGIN_PASSWORD_ERROR(600, "Username or password is wrong"),
NOT_LOGIN_ERROR(601, "Has not logged in"),
NO_PERMISSION_ERROR(602, "No permissions for this operation"),
UNINITIALIZED_AUTH_ERROR(603, "Uninitialized authorizer");
private int statusCode;
private String statusMessage;
private TSStatusType(int statusCode, String statusMessage) {
this.statusCode = statusCode;
this.statusMessage = statusMessage;
}
public int getStatusCode() {
return statusCode;
}
public String getStatusMessage() {
return statusMessage;
}
}
service-rpc/src/main/thrift/rpc.thrift
浏览文件 @ ea6daf13
......@@ -18,28 +18,17 @@
*/
namespace java org.apache.iotdb.service.rpc.thrift
// The return status code contained in each response.
enum TS_StatusCode {
SUCCESS_STATUS,
SUCCESS_WITH_INFO_STATUS,
STILL_EXECUTING_STATUS,
ERROR_STATUS,
INVALID_HANDLE_STATUS
// The return status code and message in each response.
struct TS_StatusType {
1: required i32 code
2: required string message
}
// The return status of a remote request
struct TS_Status {
1: required TS_StatusCode statusCode
// If status is SUCCESS_WITH_INFO, info_msgs may be populated with
// additional diagnostic information.
1: required TS_StatusType statusType
2: optional list<string> infoMessages
// If status is ERROR, then the following fields may be set
3: optional string sqlState // as defined in the ISO/IEF CLIENT specification
4: optional i32 errorCode // internal error code
5: optional string errorMessage
}
struct TSHandleIdentifier {
......
spark-tsfile/src/test/scala/org/apache/iotdb/tsfile/HDFSInputTest.java
浏览文件 @ ea6daf13
......@@ -64,14 +64,14 @@ public class HDFSInputTest {
@Test
public void test_read1() throws IOException {
int size = 2000;
int size = 1000;
ByteBuffer buffer = ByteBuffer.allocate(size);
Assert.assertEquals(size, in.read(buffer));
}
@Test
public void test_read2() throws IOException {
int size = 2000;
int size = 1000;
long pos = 20L;
ByteBuffer buffer = ByteBuffer.allocate(size);
Assert.assertEquals(size, in.read(buffer, pos));
......
tsfile/src/main/java/org/apache/iotdb/tsfile/file/metadata/TsDigest.java
浏览文件 @ ea6daf13
......@@ -23,10 +23,7 @@ import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.nio.ByteBuffer;
import java.util.Collections;
import java.util.HashMap;
import java.util.Map;
import java.util.Map.Entry;
import java.util.Arrays;
import org.apache.iotdb.tsfile.utils.ReadWriteIOUtils;
/**
......@@ -34,11 +31,15 @@ import org.apache.iotdb.tsfile.utils.ReadWriteIOUtils;
*/
public class TsDigest {
private Map<String, ByteBuffer> statistics;
private ByteBuffer[] statistics;
private int serializedSize = Integer.BYTES;
/**
* size of valid values in statistics. Note that some values in statistics can be null and thus
* invalid.
*/
private int validSizeOfArray = 0;
private int sizeOfList;
private int serializedSize = Integer.BYTES; // initialize for number of statistics
public TsDigest() {
// allowed to clair an empty TsDigest whose fields will be assigned later.
......@@ -66,17 +67,18 @@ public class TsDigest {
TsDigest digest = new TsDigest();
int size = ReadWriteIOUtils.readInt(inputStream);
digest.validSizeOfArray = size;
digest.serializedSize = Integer.BYTES;
if (size > 0) {
Map<String, ByteBuffer> statistics = new HashMap<>();
String key;
digest.statistics = new ByteBuffer[StatisticType.getTotalTypeNum()];
ByteBuffer value;
for (int i = 0; i < size; i++) {
key = ReadWriteIOUtils.readString(inputStream);
short n = ReadWriteIOUtils.readShort(inputStream);
value = ReadWriteIOUtils.readByteBufferWithSelfDescriptionLength(inputStream);
statistics.put(key, value);
digest.statistics[n] = value;
digest.serializedSize += Short.BYTES + Integer.BYTES + value.remaining();
}
digest.statistics = statistics;
}
} // else left digest.statistics as null
return digest;
}
......@@ -91,69 +93,56 @@ public class TsDigest {
TsDigest digest = new TsDigest();
int size = ReadWriteIOUtils.readInt(buffer);
digest.validSizeOfArray = size;
digest.serializedSize = Integer.BYTES;
if (size > 0) {
Map<String, ByteBuffer> statistics = new HashMap<>();
String key;
digest.statistics = new ByteBuffer[StatisticType.getTotalTypeNum()];
ByteBuffer value;
for (int i = 0; i < size; i++) {
key = ReadWriteIOUtils.readString(buffer);
short n = ReadWriteIOUtils.readShort(buffer);
value = ReadWriteIOUtils.readByteBufferWithSelfDescriptionLength(buffer);
statistics.put(key, value);
digest.statistics[n] = value;
digest.serializedSize += Short.BYTES + Integer.BYTES + value.remaining();
}
digest.statistics = statistics;
}
} // else left digest.statistics as null
return digest;
}
private void reCalculateSerializedSize() {
private void reCalculate() {
validSizeOfArray = 0;
serializedSize = Integer.BYTES;
if (statistics != null) {
for (Map.Entry<String, ByteBuffer> entry : statistics.entrySet()) {
serializedSize += Integer.BYTES + entry.getKey().length() + Integer.BYTES
+ entry.getValue().remaining();
for (ByteBuffer value : statistics) {
if (value != null) {
// StatisticType serialized value, byteBuffer.capacity and byteBuffer.array
serializedSize += Short.BYTES + Integer.BYTES + value.remaining();
validSizeOfArray++;
}
}
sizeOfList = statistics.size();
} else {
sizeOfList = 0;
}
}
/**
* get statistics of the current object.
*
* @return -unmodifiableMap of the current object's statistics
*/
public Map<String, ByteBuffer> getStatistics() {
if (statistics == null) {
return null;
}
return Collections.unmodifiableMap(this.statistics);
public ByteBuffer[] getStatistics() {
return statistics; //TODO unmodifiable
}
public void setStatistics(Map<String, ByteBuffer> statistics) {
this.statistics = statistics;
reCalculateSerializedSize();
}
/**
* add statistics using given param.
*
* @param key -key of the entry
* @param value -value of the entry
*/
public void addStatistics(String key, ByteBuffer value) {
if (statistics == null) {
statistics = new HashMap<>();
public void setStatistics(ByteBuffer[] statistics) throws IOException {
if (statistics != null && statistics.length != StatisticType.getTotalTypeNum()) {
throw new IOException(String.format(
"The length of array of statistics doesn't equal StatisticType.getTotalTypeNum() %d",
StatisticType.getTotalTypeNum()));
}
statistics.put(key, value);
serializedSize += Integer.BYTES + key.length() + Integer.BYTES + value.remaining();
sizeOfList++;
this.statistics = statistics;
reCalculate(); // DO NOT REMOVE THIS
}
@Override
public String toString() {
return statistics != null ? statistics.toString() : "";
return statistics != null ? Arrays.toString(statistics) : "";
}
/**
......@@ -163,20 +152,16 @@ public class TsDigest {
* @return -byte length
*/
public int serializeTo(OutputStream outputStream) throws IOException {
if ((statistics != null && sizeOfList != statistics.size()) || (statistics == null
&& sizeOfList != 0)) {
reCalculateSerializedSize();
}
int byteLen = 0;
if (statistics == null || statistics.size() == 0) {
if (validSizeOfArray == 0) {
byteLen += ReadWriteIOUtils.write(0, outputStream);
} else {
byteLen += ReadWriteIOUtils.write(statistics.size(), outputStream);
for (Map.Entry<String, ByteBuffer> entry : statistics.entrySet()) {
byteLen += ReadWriteIOUtils
.write(entry.getKey(), outputStream);
byteLen += ReadWriteIOUtils
.write(entry.getValue(), outputStream);
byteLen += ReadWriteIOUtils.write(validSizeOfArray, outputStream);
for (int i = 0; i < statistics.length; i++) {
if (statistics[i] != null) {
byteLen += ReadWriteIOUtils.write((short) i, outputStream);
byteLen += ReadWriteIOUtils.write(statistics[i], outputStream);
}
}
}
return byteLen;
......@@ -189,20 +174,16 @@ public class TsDigest {
* @return -byte length
*/
public int serializeTo(ByteBuffer buffer) {
if ((statistics != null && sizeOfList != statistics.size()) || (statistics == null
&& sizeOfList != 0)) {
reCalculateSerializedSize();
}
int byteLen = 0;
if (statistics == null || statistics.size() == 0) {
if (validSizeOfArray == 0) {
byteLen += ReadWriteIOUtils.write(0, buffer);
} else {
byteLen += ReadWriteIOUtils.write(statistics.size(), buffer);
for (Map.Entry<String, ByteBuffer> entry : statistics.entrySet()) {
byteLen += ReadWriteIOUtils.write(entry.getKey(), buffer);
byteLen += ReadWriteIOUtils
.write(entry.getValue(), buffer);
byteLen += ReadWriteIOUtils.write(validSizeOfArray, buffer);
for (int i = 0; i < statistics.length; i++) {
if (statistics[i] != null) {
byteLen += ReadWriteIOUtils.write((short) i, buffer);
byteLen += ReadWriteIOUtils.write(statistics[i], buffer);
}
}
}
return byteLen;
......@@ -214,9 +195,6 @@ public class TsDigest {
* @return -serializedSize
*/
public int getSerializedSize() {
if (statistics == null || (sizeOfList != statistics.size())) {
reCalculateSerializedSize();
}
return serializedSize;
}
......@@ -229,17 +207,44 @@ public class TsDigest {
return false;
}
TsDigest digest = (TsDigest) o;
if (serializedSize != digest.serializedSize || sizeOfList != digest.sizeOfList
|| statistics.size() != digest.statistics.size()) {
if (serializedSize != digest.serializedSize || validSizeOfArray != digest.validSizeOfArray
|| ((statistics == null) ^ (digest.statistics == null))) {
return false;
}
for (Entry<String, ByteBuffer> entry : statistics.entrySet()) {
String key = entry.getKey();
ByteBuffer value = entry.getValue();
if (!digest.statistics.containsKey(key) || !value.equals(digest.statistics.get(key))) {
return false;
if (statistics != null) {
for (int i = 0; i < statistics.length; i++) {
if ((statistics[i] == null) ^ (digest.statistics[i] == null)) {
// one is null and the other is not null
return false;
}
if (statistics[i] != null) {
if (!statistics[i].equals(digest.statistics[i])) {
return false;
}
}
}
}
return true;
}
public enum StatisticType {
min_value, max_value, first_value, last_value, sum_value;
public static int getTotalTypeNum() {
return StatisticType.values().length;
}
public static StatisticType deserialize(short i) {
return StatisticType.values()[i];
}
public static int getSerializedSize() {
return Short.BYTES;
}
public short serialize() {
return (short) this.ordinal();
}
}
}
tsfile/src/main/java/org/apache/iotdb/tsfile/file/metadata/statistics/BinaryStatistics.java
浏览文件 @ ea6daf13
......@@ -33,16 +33,16 @@ import org.apache.iotdb.tsfile.utils.ReadWriteIOUtils;
*/
public class BinaryStatistics extends Statistics<Binary> {
private Binary max = new Binary("");
private Binary min = new Binary("");
private Binary max = new Binary("");
private Binary first = new Binary("");
private double sum;// FIXME sum is meaningless
private Binary last = new Binary("");
private double sum;// FIXME sum is meaningless
@Override
public void setMinMaxFromBytes(byte[] minBytes, byte[] maxBytes) {
max = new Binary(maxBytes);
min = new Binary(minBytes);
max = new Binary(maxBytes);
}
@Override
......@@ -61,13 +61,13 @@ public class BinaryStatistics extends Statistics<Binary> {
}
@Override
public double getSum() {
return sum;
public Binary getLast() {
return last;
}
@Override
public Binary getLast() {
return last;
public double getSum() {
return sum;
}
/**
......@@ -76,15 +76,15 @@ public class BinaryStatistics extends Statistics<Binary> {
* @param min minimum value
* @param max maximum value
* @param first the first value
* @param sum sum
* @param last the last value
* @param sum sum
*/
public void initializeStats(Binary min, Binary max, Binary first, double sum, Binary last) {
private void initializeStats(Binary min, Binary max, Binary first, Binary last, double sum) {
this.min = min;
this.max = max;
this.first = first;
this.sum = sum;
this.last = last;
this.sum = sum;
}
@Override
......@@ -92,23 +92,21 @@ public class BinaryStatistics extends Statistics<Binary> {
BinaryStatistics stringStats = (BinaryStatistics) stats;
if (isEmpty) {
initializeStats(stringStats.getMin(), stringStats.getMax(), stringStats.getFirst(),
stringStats.getSum(),
stringStats.getLast());
stringStats.getLast(), stringStats.getSum());
isEmpty = false;
} else {
updateStats(stringStats.getMin(), stringStats.getMax(), stringStats.getFirst(),
stringStats.getSum(),
stringStats.getLast());
stringStats.getLast(), stringStats.getSum());
}
}
@Override
public void updateStats(Binary value) {
if (isEmpty) {
initializeStats(value, value, value, 0, value);
initializeStats(value, value, value, value, 0);
isEmpty = false;
} else {
updateStats(value, value, value, 0, value);
updateStats(value, value, value, value, 0);
isEmpty = false;
}
}
......@@ -117,17 +115,17 @@ public class BinaryStatistics extends Statistics<Binary> {
public void updateStats(Binary[] values) {
for (Binary value : values) {
if (isEmpty) {
initializeStats(value, value, value, 0, value);
initializeStats(value, value, value, value, 0);
isEmpty = false;
} else {
updateStats(value, value, value, 0, value);
updateStats(value, value, value, value, 0);
isEmpty = false;
}
}
}
private void updateStats(Binary minValue, Binary maxValue, Binary firstValue, double sum,
Binary lastValue) {
private void updateStats(Binary minValue, Binary maxValue, Binary firstValue, Binary lastValue,
double sum) {
if (minValue.compareTo(min) < 0) {
min = minValue;
}
......@@ -137,24 +135,19 @@ public class BinaryStatistics extends Statistics<Binary> {
this.last = lastValue;
}
@Override
public byte[] getMaxBytes() {
return max.getValues();
}
@Override
public byte[] getMinBytes() {
return min.getValues();
}
@Override
public byte[] getFirstBytes() {
return first.getValues();
public byte[] getMaxBytes() {
return max.getValues();
}
@Override
public byte[] getSumBytes() {
return BytesUtils.doubleToBytes(sum);
public byte[] getFirstBytes() {
return first.getValues();
}
@Override
......@@ -163,8 +156,8 @@ public class BinaryStatistics extends Statistics<Binary> {
}
@Override
public ByteBuffer getMaxBytebuffer() {
return ByteBuffer.wrap(max.getValues());
public byte[] getSumBytes() {
return BytesUtils.doubleToBytes(sum);
}
@Override
......@@ -173,13 +166,13 @@ public class BinaryStatistics extends Statistics<Binary> {
}
@Override
public ByteBuffer getFirstBytebuffer() {
return ByteBuffer.wrap(first.getValues());
public ByteBuffer getMaxBytebuffer() {
return ByteBuffer.wrap(max.getValues());
}
@Override
public ByteBuffer getSumBytebuffer() {
return ReadWriteIOUtils.getByteBuffer(sum);
public ByteBuffer getFirstBytebuffer() {
return ByteBuffer.wrap(first.getValues());
}
@Override
......@@ -187,6 +180,11 @@ public class BinaryStatistics extends Statistics<Binary> {
return ByteBuffer.wrap(last.getValues());
}
@Override
public ByteBuffer getSumBytebuffer() {
return ReadWriteIOUtils.getByteBuffer(sum);
}
@Override
public int sizeOfDatum() {
return -1;
......@@ -194,7 +192,7 @@ public class BinaryStatistics extends Statistics<Binary> {
@Override
public String toString() {
return "[max:" + max + ",min:" + min + ",first:" + first + ",sum:" + sum + ",last:" + last
return "[min:" + min + ",max:" + max + ",first:" + first + ",last:" + last + ",sum:" + sum
+ "]";
}
......
tsfile/src/main/java/org/apache/iotdb/tsfile/file/metadata/statistics/BooleanStatistics.java
浏览文件 @ ea6daf13
......@@ -31,25 +31,25 @@ import org.apache.iotdb.tsfile.utils.ReadWriteIOUtils;
*/
public class BooleanStatistics extends Statistics<Boolean> {
private boolean max;
private boolean min;
private boolean max;
private boolean first;
private double sum;
private boolean last;
private double sum;
@Override
public void setMinMaxFromBytes(byte[] minBytes, byte[] maxBytes) {
max = BytesUtils.bytesToBool(maxBytes);
min = BytesUtils.bytesToBool(minBytes);
max = BytesUtils.bytesToBool(maxBytes);
}
@Override
public void updateStats(boolean value) {
if (isEmpty) {
initializeStats(value, value, value, 0, value);
initializeStats(value, value, value, value, 0);
isEmpty = false;
} else {
updateStats(value, value, value, 0, value);
updateStats(value, value, value, value, 0);
isEmpty = false;
}
}
......@@ -58,17 +58,17 @@ public class BooleanStatistics extends Statistics<Boolean> {
public void updateStats(boolean[] values) {
for (boolean value : values) {
if (isEmpty) {
initializeStats(value, value, value, 0, value);
initializeStats(value, value, value, value, 0);
isEmpty = false;
} else {
updateStats(value, value, value, 0, value);
updateStats(value, value, value, value, 0);
isEmpty = false;
}
}
}
private void updateStats(boolean minValue, boolean maxValue, boolean firstValue, double sumValue,
boolean lastValue) {
private void updateStats(boolean minValue, boolean maxValue, boolean firstValue,
boolean lastValue, double sumValue) {
if (!minValue && min) {
min = false;
}
......@@ -78,24 +78,19 @@ public class BooleanStatistics extends Statistics<Boolean> {
this.last = lastValue;
}
@Override
public Boolean getMax() {
return max;
}
@Override
public Boolean getMin() {
return min;
}
@Override
public Boolean getFirst() {
return first;
public Boolean getMax() {
return max;
}
@Override
public double getSum() {
return sum;
public Boolean getFirst() {
return first;
}
@Override
......@@ -104,8 +99,8 @@ public class BooleanStatistics extends Statistics<Boolean> {
}
@Override
public ByteBuffer getMaxBytebuffer() {
return ReadWriteIOUtils.getByteBuffer(max);
public double getSum() {
return sum;
}
@Override
......@@ -114,13 +109,13 @@ public class BooleanStatistics extends Statistics<Boolean> {
}
@Override
public ByteBuffer getFirstBytebuffer() {
return ReadWriteIOUtils.getByteBuffer(first);
public ByteBuffer getMaxBytebuffer() {
return ReadWriteIOUtils.getByteBuffer(max);
}
@Override
public ByteBuffer getSumBytebuffer() {
return ReadWriteIOUtils.getByteBuffer(sum);
public ByteBuffer getFirstBytebuffer() {
return ReadWriteIOUtils.getByteBuffer(first);
}
@Override
......@@ -128,17 +123,21 @@ public class BooleanStatistics extends Statistics<Boolean> {
return ReadWriteIOUtils.getByteBuffer(last);
}
@Override
public ByteBuffer getSumBytebuffer() {
return ReadWriteIOUtils.getByteBuffer(sum);
}
@Override
protected void mergeStatisticsValue(Statistics<?> stats) {
BooleanStatistics boolStats = (BooleanStatistics) stats;
if (isEmpty) {
initializeStats(boolStats.getMin(), boolStats.getMax(), boolStats.getFirst(),
boolStats.getSum(),
boolStats.getLast());
boolStats.getLast(), boolStats.getSum());
isEmpty = false;
} else {
updateStats(boolStats.getMin(), boolStats.getMax(), boolStats.getFirst(), boolStats.getSum(),
boolStats.getLast());
updateStats(boolStats.getMin(), boolStats.getMax(), boolStats.getFirst(),
boolStats.getLast(), boolStats.getSum());
}
}
......@@ -148,11 +147,11 @@ public class BooleanStatistics extends Statistics<Boolean> {
* @param min min boolean
* @param max max boolean
* @param firstValue first boolean value
* @param sumValue sum value (double type)
* @param lastValue last boolean value
* @param sumValue sum value (double type)
*/
public void initializeStats(boolean min, boolean max, boolean firstValue, double sumValue,
boolean lastValue) {
private void initializeStats(boolean min, boolean max, boolean firstValue, boolean lastValue,
double sumValue) {
this.min = min;
this.max = max;
this.first = firstValue;
......@@ -160,13 +159,13 @@ public class BooleanStatistics extends Statistics<Boolean> {
}
@Override
public byte[] getMaxBytes() {
return BytesUtils.boolToBytes(max);
public byte[] getMinBytes() {
return BytesUtils.boolToBytes(min);
}
@Override
public byte[] getMinBytes() {
return BytesUtils.boolToBytes(min);
public byte[] getMaxBytes() {
return BytesUtils.boolToBytes(max);
}
@Override
......@@ -175,13 +174,13 @@ public class BooleanStatistics extends Statistics<Boolean> {
}
@Override
public byte[] getSumBytes() {
return BytesUtils.doubleToBytes(sum);
public byte[] getLastBytes() {
return BytesUtils.boolToBytes(last);
}
@Override
public byte[] getLastBytes() {
return BytesUtils.boolToBytes(last);
public byte[] getSumBytes() {
return BytesUtils.doubleToBytes(sum);
}
@Override
......@@ -191,7 +190,7 @@ public class BooleanStatistics extends Statistics<Boolean> {
@Override
public String toString() {
return "[max:" + max + ",min:" + min + ",first:" + first + ",sum:" + sum + ",last:" + last
return "[min:" + min + ",max:" + max + ",first:" + first + ",last:" + last + ",sum:" + sum
+ "]";
}
......
tsfile/src/main/java/org/apache/iotdb/tsfile/file/metadata/statistics/DoubleStatistics.java
浏览文件 @ ea6daf13
......@@ -31,16 +31,16 @@ import org.apache.iotdb.tsfile.utils.ReadWriteIOUtils;
*/
public class DoubleStatistics extends Statistics<Double> {
private double max;
private double min;
private double max;
private double first;
private double sum;
private double last;
private double sum;
@Override
public void setMinMaxFromBytes(byte[] minBytes, byte[] maxBytes) {
max = BytesUtils.bytesToDouble(maxBytes);
min = BytesUtils.bytesToDouble(minBytes);
max = BytesUtils.bytesToDouble(maxBytes);
}
@Override
......@@ -65,8 +65,8 @@ public class DoubleStatistics extends Statistics<Double> {
}
}
private void updateStats(double minValue, double maxValue, double firstValue, double sumValue,
double lastValue) {
private void updateStats(double minValue, double maxValue, double firstValue, double lastValue,
double sumValue) {
if (minValue < min) {
min = minValue;
}
......@@ -78,13 +78,13 @@ public class DoubleStatistics extends Statistics<Double> {
}
@Override
public Double getMax() {
return max;
public Double getMin() {
return min;
}
@Override
public Double getMin() {
return min;
public Double getMax() {
return max;
}
@Override
......@@ -93,13 +93,13 @@ public class DoubleStatistics extends Statistics<Double> {
}
@Override
public double getSum() {
return sum;
public Double getLast() {
return last;
}
@Override
public Double getLast() {
return last;
public double getSum() {
return sum;
}
@Override
......@@ -107,13 +107,11 @@ public class DoubleStatistics extends Statistics<Double> {
DoubleStatistics doubleStats = (DoubleStatistics) stats;
if (this.isEmpty) {
initializeStats(doubleStats.getMin(), doubleStats.getMax(), doubleStats.getFirst(),
doubleStats.getSum(),
doubleStats.getLast());
doubleStats.getLast(), doubleStats.getSum());
isEmpty = false;
} else {
updateStats(doubleStats.getMin(), doubleStats.getMax(), doubleStats.getFirst(),
doubleStats.getSum(),
doubleStats.getLast());
doubleStats.getLast(), doubleStats.getSum());
}
}
......@@ -124,20 +122,15 @@ public class DoubleStatistics extends Statistics<Double> {
* @param min min value
* @param max max value
* @param first the first value
* @param sum sum value
* @param last the last value
* @param sum sum value
*/
public void initializeStats(double min, double max, double first, double sum, double last) {
private void initializeStats(double min, double max, double first, double last, double sum) {
this.min = min;
this.max = max;
this.first = first;
this.sum = sum;
this.last = last;
}
@Override
public byte[] getMaxBytes() {
return BytesUtils.doubleToBytes(max);
this.sum = sum;
}
@Override
......@@ -146,13 +139,13 @@ public class DoubleStatistics extends Statistics<Double> {
}
@Override
public byte[] getFirstBytes() {
return BytesUtils.doubleToBytes(first);
public byte[] getMaxBytes() {
return BytesUtils.doubleToBytes(max);
}
@Override
public byte[] getSumBytes() {
return BytesUtils.doubleToBytes(sum);
public byte[] getFirstBytes() {
return BytesUtils.doubleToBytes(first);
}
@Override
......@@ -161,8 +154,8 @@ public class DoubleStatistics extends Statistics<Double> {
}
@Override
public ByteBuffer getMaxBytebuffer() {
return ReadWriteIOUtils.getByteBuffer(max);
public byte[] getSumBytes() {
return BytesUtils.doubleToBytes(sum);
}
@Override
......@@ -171,13 +164,13 @@ public class DoubleStatistics extends Statistics<Double> {
}
@Override
public ByteBuffer getFirstBytebuffer() {
return ReadWriteIOUtils.getByteBuffer(first);
public ByteBuffer getMaxBytebuffer() {
return ReadWriteIOUtils.getByteBuffer(max);
}
@Override
public ByteBuffer getSumBytebuffer() {
return ReadWriteIOUtils.getByteBuffer(sum);
public ByteBuffer getFirstBytebuffer() {
return ReadWriteIOUtils.getByteBuffer(first);
}
@Override
......@@ -185,6 +178,11 @@ public class DoubleStatistics extends Statistics<Double> {
return ReadWriteIOUtils.getByteBuffer(last);
}
@Override
public ByteBuffer getSumBytebuffer() {
return ReadWriteIOUtils.getByteBuffer(sum);
}
@Override
public int sizeOfDatum() {
return 8;
......@@ -192,7 +190,7 @@ public class DoubleStatistics extends Statistics<Double> {
@Override
public String toString() {
return "[max:" + max + ",min:" + min + ",first:" + first + ",sum:" + sum + ",last:" + last
return "[min:" + min + ",max:" + max + ",first:" + first + ",last:" + last + ",sum:" + sum
+ "]";
}
......
tsfile/src/main/java/org/apache/iotdb/tsfile/file/metadata/statistics/FloatStatistics.java
浏览文件 @ ea6daf13
......@@ -31,16 +31,16 @@ import org.apache.iotdb.tsfile.utils.ReadWriteIOUtils;
*/
public class FloatStatistics extends Statistics<Float> {
private float max;
private float min;
private float max;
private float first;
private double sum;
private float last;
@Override
public void setMinMaxFromBytes(byte[] minBytes, byte[] maxBytes) {
max = BytesUtils.bytesToFloat(maxBytes);
min = BytesUtils.bytesToFloat(minBytes);
max = BytesUtils.bytesToFloat(maxBytes);
}
@Override
......@@ -65,8 +65,8 @@ public class FloatStatistics extends Statistics<Float> {
}
}
private void updateStats(float minValue, float maxValue, float firstValue,
double sumValue, float last) {
private void updateStats(float minValue, float maxValue, float firstValue, float last,
double sumValue) {
if (minValue < min) {
min = minValue;
}
......@@ -78,13 +78,13 @@ public class FloatStatistics extends Statistics<Float> {
}
@Override
public Float getMax() {
return max;
public Float getMin() {
return min;
}
@Override
public Float getMin() {
return min;
public Float getMax() {
return max;
}
@Override
......@@ -93,13 +93,13 @@ public class FloatStatistics extends Statistics<Float> {
}
@Override
public double getSum() {
return sum;
public Float getLast() {
return last;
}
@Override
public Float getLast() {
return last;
public double getSum() {
return sum;
}
@Override
......@@ -107,28 +107,21 @@ public class FloatStatistics extends Statistics<Float> {
FloatStatistics floatStats = (FloatStatistics) stats;
if (isEmpty) {
initializeStats(floatStats.getMin(), floatStats.getMax(), floatStats.getFirst(),
floatStats.getSum(),
floatStats.getLast());
floatStats.getLast(), floatStats.getSum());
isEmpty = false;
} else {
updateStats(floatStats.getMin(), floatStats.getMax(), floatStats.getFirst(),
floatStats.getSum(),
floatStats.getLast());
floatStats.getLast(), floatStats.getSum());
}
}
public void initializeStats(float min, float max, float first, double sum, float last) {
private void initializeStats(float min, float max, float first, float last, double sum) {
this.min = min;
this.max = max;
this.first = first;
this.sum = sum;
this.last = last;
}
@Override
public byte[] getMaxBytes() {
return BytesUtils.floatToBytes(max);
this.sum = sum;
}
@Override
......@@ -137,13 +130,13 @@ public class FloatStatistics extends Statistics<Float> {
}
@Override
public byte[] getFirstBytes() {
return BytesUtils.floatToBytes(first);
public byte[] getMaxBytes() {
return BytesUtils.floatToBytes(max);
}
@Override
public byte[] getSumBytes() {
return BytesUtils.doubleToBytes(sum);
public byte[] getFirstBytes() {
return BytesUtils.floatToBytes(first);
}
@Override
......@@ -152,8 +145,8 @@ public class FloatStatistics extends Statistics<Float> {
}
@Override
public ByteBuffer getMaxBytebuffer() {
return ReadWriteIOUtils.getByteBuffer(max);
public byte[] getSumBytes() {
return BytesUtils.doubleToBytes(sum);
}
@Override
......@@ -162,13 +155,13 @@ public class FloatStatistics extends Statistics<Float> {
}
@Override
public ByteBuffer getFirstBytebuffer() {
return ReadWriteIOUtils.getByteBuffer(first);
public ByteBuffer getMaxBytebuffer() {
return ReadWriteIOUtils.getByteBuffer(max);
}
@Override
public ByteBuffer getSumBytebuffer() {
return ReadWriteIOUtils.getByteBuffer(sum);
public ByteBuffer getFirstBytebuffer() {
return ReadWriteIOUtils.getByteBuffer(first);
}
@Override
......@@ -176,6 +169,11 @@ public class FloatStatistics extends Statistics<Float> {
return ReadWriteIOUtils.getByteBuffer(last);
}
@Override
public ByteBuffer getSumBytebuffer() {
return ReadWriteIOUtils.getByteBuffer(sum);
}
@Override
public int sizeOfDatum() {
return 4;
......@@ -183,8 +181,8 @@ public class FloatStatistics extends Statistics<Float> {
@Override
public String toString() {
return "[max:" + max + ",min:" + min + ",first:"
+ first + ",sum:" + sum + ",last:" + last + "]";
return "[min:" + min + ",max:" + max + ",first:" + first + ",last:" + last + ",sum:" + sum
+ "]";
}
@Override
......
tsfile/src/main/java/org/apache/iotdb/tsfile/file/metadata/statistics/IntegerStatistics.java
浏览文件 @ ea6daf13
......@@ -31,16 +31,16 @@ import org.apache.iotdb.tsfile.utils.ReadWriteIOUtils;
*/
public class IntegerStatistics extends Statistics<Integer> {
private int max;
private int min;
private int max;
private int first;
private double sum;
private int last;
private double sum;
@Override
public void setMinMaxFromBytes(byte[] minBytes, byte[] maxBytes) {
max = BytesUtils.bytesToInt(maxBytes);
min = BytesUtils.bytesToInt(minBytes);
max = BytesUtils.bytesToInt(maxBytes);
}
@Override
......@@ -67,8 +67,8 @@ public class IntegerStatistics extends Statistics<Integer> {
}
}
private void updateStats(int minValue, int maxValue,
int firstValue, double sumValue, int lastValue) {
private void updateStats(int minValue, int maxValue, int firstValue, int lastValue,
double sumValue) {
// TODO: unused parameter
if (minValue < min) {
min = minValue;
......@@ -81,13 +81,13 @@ public class IntegerStatistics extends Statistics<Integer> {
}
@Override
public Integer getMax() {
return max;
public Integer getMin() {
return min;
}
@Override
public Integer getMin() {
return min;
public Integer getMax() {
return max;
}
@Override
......@@ -96,40 +96,35 @@ public class IntegerStatistics extends Statistics<Integer> {
}
@Override
public double getSum() {
return sum;
public Integer getLast() {
return last;
}
@Override
public Integer getLast() {
return last;
public double getSum() {
return sum;
}
@Override
protected void mergeStatisticsValue(Statistics<?> stats) {
IntegerStatistics intStats = (IntegerStatistics) stats;
if (isEmpty) {
initializeStats(intStats.getMin(), intStats.getMax(), intStats.getFirst(), intStats.getSum(),
intStats.getLast());
initializeStats(intStats.getMin(), intStats.getMax(), intStats.getFirst(), intStats.getLast(),
intStats.getSum());
isEmpty = false;
} else {
updateStats(intStats.getMin(), intStats.getMax(), intStats.getFirst(), intStats.getSum(),
intStats.getLast());
updateStats(intStats.getMin(), intStats.getMax(), intStats.getFirst(), intStats.getLast(),
intStats.getSum());
}
}
void initializeStats(int min, int max, int first, double sum, int last) {
private void initializeStats(int min, int max, int first, int last, double sum) {
this.min = min;
this.max = max;
this.first = first;
this.sum = sum;
this.last = last;
}
@Override
public ByteBuffer getMaxBytebuffer() {
return ReadWriteIOUtils.getByteBuffer(max);
this.sum = sum;
}
@Override
......@@ -138,13 +133,13 @@ public class IntegerStatistics extends Statistics<Integer> {
}
@Override
public ByteBuffer getFirstBytebuffer() {
return ReadWriteIOUtils.getByteBuffer(first);
public ByteBuffer getMaxBytebuffer() {
return ReadWriteIOUtils.getByteBuffer(max);
}
@Override
public ByteBuffer getSumBytebuffer() {
return ReadWriteIOUtils.getByteBuffer(sum);
public ByteBuffer getFirstBytebuffer() {
return ReadWriteIOUtils.getByteBuffer(first);
}
@Override
......@@ -153,8 +148,8 @@ public class IntegerStatistics extends Statistics<Integer> {
}
@Override
public byte[] getMaxBytes() {
return BytesUtils.intToBytes(max);
public ByteBuffer getSumBytebuffer() {
return ReadWriteIOUtils.getByteBuffer(sum);
}
@Override
......@@ -163,13 +158,13 @@ public class IntegerStatistics extends Statistics<Integer> {
}
@Override
public byte[] getFirstBytes() {
return BytesUtils.intToBytes(first);
public byte[] getMaxBytes() {
return BytesUtils.intToBytes(max);
}
@Override
public byte[] getSumBytes() {
return BytesUtils.doubleToBytes(sum);
public byte[] getFirstBytes() {
return BytesUtils.intToBytes(first);
}
@Override
......@@ -177,6 +172,11 @@ public class IntegerStatistics extends Statistics<Integer> {
return BytesUtils.intToBytes(last);
}
@Override
public byte[] getSumBytes() {
return BytesUtils.doubleToBytes(sum);
}
@Override
public int sizeOfDatum() {
return 4;
......@@ -184,8 +184,8 @@ public class IntegerStatistics extends Statistics<Integer> {
@Override
public String toString() {
return "[max:" + max + ",min:" + min + ",first:"
+ first + ",sum:" + sum + ",last:" + last + "]";
return "[min:" + min + ",max:" + max + ",first:" + first + ",last:" + last + ",sum:" + sum
+ "]";
}
@Override
......
tsfile/src/main/java/org/apache/iotdb/tsfile/file/metadata/statistics/LongStatistics.java
浏览文件 @ ea6daf13
......@@ -31,16 +31,16 @@ import org.apache.iotdb.tsfile.utils.ReadWriteIOUtils;
*/
public class LongStatistics extends Statistics<Long> {
private long max;
private long min;
private long max;
private long first;
private double sum;
private long last;
private double sum;
@Override
public void setMinMaxFromBytes(byte[] minBytes, byte[] maxBytes) {
max = BytesUtils.bytesToLong(maxBytes);
min = BytesUtils.bytesToLong(minBytes);
max = BytesUtils.bytesToLong(maxBytes);
}
@Override
......@@ -59,13 +59,13 @@ public class LongStatistics extends Statistics<Long> {
}
@Override
public double getSum() {
return sum;
public Long getLast() {
return last;
}
@Override
public Long getLast() {
return last;
public double getSum() {
return sum;
}
@Override
......@@ -90,8 +90,8 @@ public class LongStatistics extends Statistics<Long> {
}
}
private void updateStats(long minValue, long maxValue, long firstValue, double sumValue,
long lastValue) {
private void updateStats(long minValue, long maxValue, long firstValue, long lastValue,
double sumValue) {
if (minValue < min) {
min = minValue;
}
......@@ -117,27 +117,21 @@ public class LongStatistics extends Statistics<Long> {
LongStatistics longStats = (LongStatistics) stats;
if (isEmpty) {
initializeStats(longStats.getMin(), longStats.getMax(), longStats.getFirst(),
longStats.getSum(),
longStats.getLast());
longStats.getLast(), longStats.getSum());
isEmpty = false;
} else {
updateStats(longStats.getMin(), longStats.getMax(), longStats.getFirst(), longStats.getSum(),
longStats.getLast());
updateStats(longStats.getMin(), longStats.getMax(), longStats.getFirst(), longStats.getLast(),
longStats.getSum());
}
}
void initializeStats(long min, long max, long firstValue, double sum, long last) {
private void initializeStats(long min, long max, long firstValue, long last, double sum) {
this.min = min;
this.max = max;
this.first = firstValue;
this.sum += sum;
this.last = last;
}
@Override
public byte[] getMaxBytes() {
return BytesUtils.longToBytes(max);
this.sum += sum;
}
@Override
......@@ -146,13 +140,13 @@ public class LongStatistics extends Statistics<Long> {
}
@Override
public byte[] getFirstBytes() {
return BytesUtils.longToBytes(first);
public byte[] getMaxBytes() {
return BytesUtils.longToBytes(max);
}
@Override
public byte[] getSumBytes() {
return BytesUtils.doubleToBytes(sum);
public byte[] getFirstBytes() {
return BytesUtils.longToBytes(first);
}
@Override
......@@ -161,8 +155,8 @@ public class LongStatistics extends Statistics<Long> {
}
@Override
public ByteBuffer getMaxBytebuffer() {
return ReadWriteIOUtils.getByteBuffer(max);
public byte[] getSumBytes() {
return BytesUtils.doubleToBytes(sum);
}
@Override
......@@ -171,13 +165,13 @@ public class LongStatistics extends Statistics<Long> {
}
@Override
public ByteBuffer getFirstBytebuffer() {
return ReadWriteIOUtils.getByteBuffer(first);
public ByteBuffer getMaxBytebuffer() {
return ReadWriteIOUtils.getByteBuffer(max);
}
@Override
public ByteBuffer getSumBytebuffer() {
return ReadWriteIOUtils.getByteBuffer(sum);
public ByteBuffer getFirstBytebuffer() {
return ReadWriteIOUtils.getByteBuffer(first);
}
@Override
......@@ -185,9 +179,14 @@ public class LongStatistics extends Statistics<Long> {
return ReadWriteIOUtils.getByteBuffer(last);
}
@Override
public ByteBuffer getSumBytebuffer() {
return ReadWriteIOUtils.getByteBuffer(sum);
}
@Override
public String toString() {
return "[max:" + max + ",min:" + min + ",first:" + first + ",sum:" + sum + ",last:" + last
return "[min:" + min + ",max:" + max + ",first:" + first + ",last:" + last + ",sum:" + sum
+ "]";
}
......
tsfile/src/main/java/org/apache/iotdb/tsfile/file/metadata/statistics/Statistics.java
浏览文件 @ ea6daf13
......@@ -105,30 +105,30 @@ public abstract class Statistics<T> {
public abstract T getFirst();
public abstract double getSum();
public abstract T getLast();
public abstract byte[] getMaxBytes();
public abstract double getSum();
public abstract byte[] getMinBytes();
public abstract byte[] getFirstBytes();
public abstract byte[] getMaxBytes();
public abstract byte[] getSumBytes();
public abstract byte[] getFirstBytes();
public abstract byte[] getLastBytes();
public abstract ByteBuffer getMaxBytebuffer();
public abstract byte[] getSumBytes();
public abstract ByteBuffer getMinBytebuffer();
public abstract ByteBuffer getFirstBytebuffer();
public abstract ByteBuffer getMaxBytebuffer();
public abstract ByteBuffer getSumBytebuffer();
public abstract ByteBuffer getFirstBytebuffer();
public abstract ByteBuffer getLastBytebuffer();
public abstract ByteBuffer getSumBytebuffer();
/**
* merge parameter to this statistic. Including
*
......@@ -262,7 +262,7 @@ public abstract class Statistics<T> {
} else if (sizeOfDatum() != -1) {
return sizeOfDatum() * 4 + 8;
} else {
return 4 * Integer.BYTES + getMaxBytes().length + getMinBytes().length
return 4 * Integer.BYTES + getMinBytes().length + getMaxBytes().length
+ getFirstBytes().length
+ getLastBytes().length + getSumBytes().length;
}
......
tsfile/src/main/java/org/apache/iotdb/tsfile/read/TsFileSequenceReader.java
浏览文件 @ ea6daf13
......@@ -31,7 +31,6 @@ import java.util.List;
import java.util.Map;
import java.util.Optional;
import org.apache.iotdb.tsfile.common.conf.TSFileConfig;
import org.apache.iotdb.tsfile.common.constant.StatisticConstant;
import org.apache.iotdb.tsfile.compress.IUnCompressor;
import org.apache.iotdb.tsfile.file.MetaMarker;
import org.apache.iotdb.tsfile.file.footer.ChunkGroupFooter;
......@@ -42,6 +41,7 @@ import org.apache.iotdb.tsfile.file.metadata.ChunkMetaData;
import org.apache.iotdb.tsfile.file.metadata.TsDeviceMetadata;
import org.apache.iotdb.tsfile.file.metadata.TsDeviceMetadataIndex;
import org.apache.iotdb.tsfile.file.metadata.TsDigest;
import org.apache.iotdb.tsfile.file.metadata.TsDigest.StatisticType;
import org.apache.iotdb.tsfile.file.metadata.TsFileMetaData;
import org.apache.iotdb.tsfile.file.metadata.enums.CompressionType;
import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
......@@ -596,16 +596,21 @@ public class TsFileSequenceReader implements AutoCloseable {
this.skipPageData(pageHeader);
}
currentChunk = new ChunkMetaData(measurementID, dataType, fileOffsetOfChunk,
startTimeOfChunk, endTimeOfChunk);
startTimeOfChunk, endTimeOfChunk);
currentChunk.setNumOfPoints(numOfPoints);
Map<String, ByteBuffer> statisticsMap = new HashMap<>();
statisticsMap.put(StatisticConstant.MAX_VALUE, ByteBuffer.wrap(chunkStatistics.getMaxBytes()));
statisticsMap.put(StatisticConstant.MIN_VALUE, ByteBuffer.wrap(chunkStatistics.getMinBytes()));
statisticsMap.put(StatisticConstant.FIRST, ByteBuffer.wrap(chunkStatistics.getFirstBytes()));
statisticsMap.put(StatisticConstant.SUM, ByteBuffer.wrap(chunkStatistics.getSumBytes()));
statisticsMap.put(StatisticConstant.LAST, ByteBuffer.wrap(chunkStatistics.getLastBytes()));
ByteBuffer[] statisticsArray = new ByteBuffer[StatisticType.getTotalTypeNum()];
statisticsArray[StatisticType.min_value.ordinal()] = ByteBuffer
.wrap(chunkStatistics.getMinBytes());
statisticsArray[StatisticType.max_value.ordinal()] = ByteBuffer
.wrap(chunkStatistics.getMaxBytes());
statisticsArray[StatisticType.first_value.ordinal()] = ByteBuffer
.wrap(chunkStatistics.getFirstBytes());
statisticsArray[StatisticType.last_value.ordinal()] = ByteBuffer
.wrap(chunkStatistics.getLastBytes());
statisticsArray[StatisticType.sum_value.ordinal()] = ByteBuffer
.wrap(chunkStatistics.getSumBytes());
TsDigest tsDigest = new TsDigest();
tsDigest.setStatistics(statisticsMap);
tsDigest.setStatistics(statisticsArray);
currentChunk.setDigest(tsDigest);
chunks.add(currentChunk);
numOfPoints = 0;
......
tsfile/src/main/java/org/apache/iotdb/tsfile/read/filter/DigestForFilter.java
浏览文件 @ ea6daf13
......@@ -100,10 +100,24 @@ public class DigestForFilter {
return maxTime;
}
public boolean isMinValueNull() {
return minValue == null;
}
public boolean isMaxValueNull() {
return maxValue == null;
}
/**
* Note check isMinValueNull before its usage
*/
public <T extends Comparable<T>> T getMinValue() {
return getValue(minValue);
}
/**
* Note check isMaxValueNull before its usage
*/
public <T extends Comparable<T>> T getMaxValue() {
return getValue(maxValue);
}
......
tsfile/src/main/java/org/apache/iotdb/tsfile/read/filter/operator/Eq.java
浏览文件 @ ea6daf13
......@@ -41,6 +41,9 @@ public class Eq<T extends Comparable<T>> extends UnaryFilter<T> {
if (filterType == FilterType.TIME_FILTER) {
return ((Long) value) >= digest.getMinTime() && ((Long) value) <= digest.getMaxTime();
} else {
if (digest.isMinValueNull() || digest.isMaxValueNull()) {
return true;
}
return value.compareTo(digest.getMinValue()) >= 0
&& value.compareTo(digest.getMaxValue()) <= 0;
}
......
tsfile/src/main/java/org/apache/iotdb/tsfile/read/filter/operator/Gt.java
浏览文件 @ ea6daf13
......@@ -41,6 +41,9 @@ public class Gt<T extends Comparable<T>> extends UnaryFilter<T> {
if (filterType == FilterType.TIME_FILTER) {
return ((Long) value) < digest.getMaxTime();
} else {
if (digest.isMaxValueNull()) {
return true;
}
return value.compareTo(digest.getMaxValue()) < 0;
}
}
......
tsfile/src/main/java/org/apache/iotdb/tsfile/read/filter/operator/GtEq.java
浏览文件 @ ea6daf13
......@@ -41,6 +41,9 @@ public class GtEq<T extends Comparable<T>> extends UnaryFilter<T> {
if (filterType == FilterType.TIME_FILTER) {
return ((Long) value) <= digest.getMaxTime();
} else {
if (digest.isMaxValueNull()) {
return true;
}
return value.compareTo(digest.getMaxValue()) <= 0;
}
}
......
tsfile/src/main/java/org/apache/iotdb/tsfile/read/filter/operator/Lt.java
浏览文件 @ ea6daf13
......@@ -41,6 +41,9 @@ public class Lt<T extends Comparable<T>> extends UnaryFilter<T> {
if (filterType == FilterType.TIME_FILTER) {
return ((Long) value) > digest.getMinTime();
} else {
if (digest.isMinValueNull()) {
return true;
}
return value.compareTo(digest.getMinValue()) > 0;
}
}
......
tsfile/src/main/java/org/apache/iotdb/tsfile/read/filter/operator/LtEq.java
浏览文件 @ ea6daf13
......@@ -41,6 +41,9 @@ public class LtEq<T extends Comparable<T>> extends UnaryFilter<T> {
if (filterType == FilterType.TIME_FILTER) {
return ((Long) value) >= digest.getMinTime();
} else {
if (digest.isMinValueNull()) {
return true;
}
return value.compareTo(digest.getMinValue()) >= 0;
}
}
......
tsfile/src/main/java/org/apache/iotdb/tsfile/read/filter/operator/NotEq.java
浏览文件 @ ea6daf13
......@@ -41,6 +41,9 @@ public class NotEq<T extends Comparable<T>> extends UnaryFilter<T> {
if (filterType == FilterType.TIME_FILTER) {
return !(((Long) value) == digest.getMinTime() && (Long) value == digest.getMaxTime());
} else {
if (digest.isMinValueNull() || digest.isMaxValueNull()) {
return true;
}
return !(value.compareTo(digest.getMinValue()) == 0
&& value.compareTo(digest.getMaxValue()) == 0);
}
......
tsfile/src/main/java/org/apache/iotdb/tsfile/read/reader/series/FileSeriesReaderWithFilter.java
浏览文件 @ ea6daf13
......@@ -19,9 +19,10 @@
package org.apache.iotdb.tsfile.read.reader.series;
import java.io.IOException;
import java.nio.ByteBuffer;
import java.util.List;
import org.apache.iotdb.tsfile.common.constant.StatisticConstant;
import org.apache.iotdb.tsfile.file.metadata.ChunkMetaData;
import org.apache.iotdb.tsfile.file.metadata.TsDigest.StatisticType;
import org.apache.iotdb.tsfile.read.common.Chunk;
import org.apache.iotdb.tsfile.read.controller.IChunkLoader;
import org.apache.iotdb.tsfile.read.filter.DigestForFilter;
......@@ -50,11 +51,16 @@ public class FileSeriesReaderWithFilter extends FileSeriesReader {
@Override
protected boolean chunkSatisfied(ChunkMetaData chunkMetaData) {
ByteBuffer minValue = null;
ByteBuffer maxValue = null;
ByteBuffer[] statistics = chunkMetaData.getDigest().getStatistics();
if (statistics != null) {
minValue = statistics[StatisticType.min_value.ordinal()]; // note still CAN be null
maxValue = statistics[StatisticType.max_value.ordinal()]; // note still CAN be null
}
DigestForFilter digest = new DigestForFilter(chunkMetaData.getStartTime(),
chunkMetaData.getEndTime(),
chunkMetaData.getDigest().getStatistics().get(StatisticConstant.MIN_VALUE),
chunkMetaData.getDigest().getStatistics().get(StatisticConstant.MAX_VALUE),
chunkMetaData.getTsDataType());
chunkMetaData.getEndTime(), minValue, maxValue, chunkMetaData.getTsDataType());
return filter.satisfy(digest);
}
......
tsfile/src/main/java/org/apache/iotdb/tsfile/write/writer/TsFileIOWriter.java
浏览文件 @ ea6daf13
......@@ -21,13 +21,26 @@ package org.apache.iotdb.tsfile.write.writer;
import java.io.File;
import java.io.IOException;
import java.nio.ByteBuffer;
import java.util.*;
import java.util.ArrayList;
import java.util.Collections;
import java.util.HashMap;
import java.util.Iterator;
import java.util.List;
import java.util.Map;
import java.util.TreeMap;
import org.apache.iotdb.tsfile.common.conf.TSFileConfig;
import org.apache.iotdb.tsfile.common.constant.StatisticConstant;
import org.apache.iotdb.tsfile.file.MetaMarker;
import org.apache.iotdb.tsfile.file.footer.ChunkGroupFooter;
import org.apache.iotdb.tsfile.file.header.ChunkHeader;
import org.apache.iotdb.tsfile.file.metadata.*;
import org.apache.iotdb.tsfile.file.metadata.ChunkGroupMetaData;
import org.apache.iotdb.tsfile.file.metadata.ChunkMetaData;
import org.apache.iotdb.tsfile.file.metadata.TsDeviceMetadata;
import org.apache.iotdb.tsfile.file.metadata.TsDeviceMetadataIndex;
import org.apache.iotdb.tsfile.file.metadata.TsDigest;
import org.apache.iotdb.tsfile.file.metadata.TsDigest.StatisticType;
import org.apache.iotdb.tsfile.file.metadata.TsFileMetaData;
import org.apache.iotdb.tsfile.file.metadata.enums.CompressionType;
import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
import org.apache.iotdb.tsfile.file.metadata.enums.TSEncoding;
......@@ -37,8 +50,8 @@ import org.apache.iotdb.tsfile.read.common.Path;
import org.apache.iotdb.tsfile.utils.BytesUtils;
import org.apache.iotdb.tsfile.utils.PublicBAOS;
import org.apache.iotdb.tsfile.utils.ReadWriteIOUtils;
import org.apache.iotdb.tsfile.write.schema.Schema;
import org.apache.iotdb.tsfile.write.schema.MeasurementSchema;
import org.apache.iotdb.tsfile.write.schema.Schema;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
......@@ -91,7 +104,6 @@ public class TsFileIOWriter {
* for writing a new tsfile.
*
* @param output be used to output written data
* @throws IOException
*/
public TsFileIOWriter(TsFileOutput output) throws IOException {
this.out = output;
......@@ -137,7 +149,8 @@ public class TsFileIOWriter {
*/
public void startChunkGroup(String deviceId) throws IOException {
LOG.debug("start chunk group:{}, file position {}", deviceId, out.getPosition());
currentChunkGroupMetaData = new ChunkGroupMetaData(deviceId, new ArrayList<>(), out.getPosition());
currentChunkGroupMetaData = new ChunkGroupMetaData(deviceId, new ArrayList<>(),
out.getPosition());
}
/**
......@@ -145,7 +158,8 @@ public class TsFileIOWriter {
*/
public void endChunkGroup(long version) throws IOException {
long dataSize = out.getPosition() - currentChunkGroupMetaData.getStartOffsetOfChunkGroup();
ChunkGroupFooter chunkGroupFooter = new ChunkGroupFooter(currentChunkGroupMetaData.getDeviceID(),
ChunkGroupFooter chunkGroupFooter = new ChunkGroupFooter(
currentChunkGroupMetaData.getDeviceID(),
dataSize, currentChunkGroupMetaData.getChunkMetaDataList().size());
chunkGroupFooter.serializeTo(out.wrapAsStream());
currentChunkGroupMetaData.setEndOffsetOfChunkGroup(out.getPosition());
......@@ -183,17 +197,19 @@ public class TsFileIOWriter {
header.serializeTo(out.wrapAsStream());
LOG.debug("finish series chunk:{} header, file position {}", header, out.getPosition());
Map<String, ByteBuffer> statisticsMap = new HashMap<>();
// TODO add your statistics
statisticsMap.put(StatisticConstant.MAX_VALUE, ByteBuffer.wrap(statistics.getMaxBytes()));
statisticsMap.put(StatisticConstant.MIN_VALUE, ByteBuffer.wrap(statistics.getMinBytes()));
statisticsMap.put(StatisticConstant.FIRST, ByteBuffer.wrap(statistics.getFirstBytes()));
statisticsMap.put(StatisticConstant.SUM, ByteBuffer.wrap(statistics.getSumBytes()));
statisticsMap.put(StatisticConstant.LAST, ByteBuffer.wrap(statistics.getLastBytes()));
ByteBuffer[] statisticsArray = new ByteBuffer[StatisticType.getTotalTypeNum()];
statisticsArray[StatisticType.max_value.ordinal()] = ByteBuffer.wrap(statistics.getMaxBytes());
statisticsArray[StatisticType.min_value.ordinal()] = ByteBuffer.wrap(statistics.getMinBytes());
statisticsArray[StatisticType.first_value.ordinal()] = ByteBuffer
.wrap(statistics.getFirstBytes());
statisticsArray[StatisticType.last_value.ordinal()] = ByteBuffer
.wrap(statistics.getLastBytes());
statisticsArray[StatisticType.sum_value.ordinal()] = ByteBuffer.wrap(statistics.getSumBytes());
TsDigest tsDigest = new TsDigest();
tsDigest.setStatistics(statisticsMap);
tsDigest.setStatistics(statisticsArray);
currentChunkMetaData.setDigest(tsDigest);
......@@ -338,7 +354,6 @@ public class TsFileIOWriter {
}
/**
* get chunkGroupMetaDataList.
*
......@@ -361,7 +376,12 @@ public class TsFileIOWriter {
}
/**
<<<<<<< HEAD
* close the outputStream or file channel without writing FileMetadata.
=======
* close the outputStream or file channel without writing FileMetadata. This is just used for
* Testing.
>>>>>>> master
*/
public void close() throws IOException {
canWrite = false;
......@@ -371,6 +391,7 @@ public class TsFileIOWriter {
void writeSeparatorMaskForTest() throws IOException {
out.write(new byte[]{MetaMarker.SEPARATOR});
}
void writeChunkMaskForTest() throws IOException {
out.write(new byte[]{MetaMarker.CHUNK_HEADER});
}
......
tsfile/src/test/java/org/apache/iotdb/tsfile/file/metadata/ChunkGroupMetaDataTest.java
浏览文件 @ ea6daf13
......@@ -18,11 +18,13 @@
*/
package org.apache.iotdb.tsfile.file.metadata;
import java.io.*;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.stream.Collectors;
import org.apache.iotdb.tsfile.exception.write.WriteProcessException;
import org.apache.iotdb.tsfile.file.MetaMarker;
import org.apache.iotdb.tsfile.file.header.ChunkHeader;
......@@ -87,7 +89,7 @@ public class ChunkGroupMetaDataTest {
}
int offsetListIndex = 0;
List<TsDeviceMetadataIndex> deviceMetadataIndexList = metaData.getDeviceMap().values().stream()
.sorted((x, y) -> (int) (x.getOffset() - y.getOffset())).collect(Collectors.toList());
.sorted((x, y) -> (int) (x.getOffset() - y.getOffset())).collect(Collectors.toList());
for (TsDeviceMetadataIndex index : deviceMetadataIndexList) {
TsDeviceMetadata deviceMetadata = reader.readTsDeviceMetaData(index);
List<ChunkGroupMetaData> chunkGroupMetaDataList = deviceMetadata.getChunkGroupMetaDataList();
......
tsfile/src/test/java/org/apache/iotdb/tsfile/file/metadata/ChunkMetaDataTest.java
浏览文件 @ ea6daf13
......@@ -52,7 +52,7 @@ public class ChunkMetaDataTest {
}
@Test
public void testWriteIntoFile() throws IOException {
public void testWriteIntoFile() {
ChunkMetaData metaData = TestHelper.createSimpleTimeSeriesChunkMetaData();
serialized(metaData);
ChunkMetaData readMetaData = deSerialized();
......@@ -60,6 +60,15 @@ public class ChunkMetaDataTest {
serialized(readMetaData);
}
@Test
public void testWriteIntoFile2() throws IOException {
ChunkMetaData metaData = TestHelper.createNotCompleteSimpleTimeSeriesChunkMetaData();
serialized(metaData);
ChunkMetaData readMetaData = deSerialized();
Utils.isTimeSeriesChunkMetadataEqual(metaData, readMetaData);
serialized(readMetaData);
}
private ChunkMetaData deSerialized() {
FileInputStream fis = null;
ChunkMetaData metaData = null;
......
tsfile/src/test/java/org/apache/iotdb/tsfile/file/metadata/utils/TestHelper.java
浏览文件 @ ea6daf13
......@@ -18,6 +18,7 @@
*/
package org.apache.iotdb.tsfile.file.metadata.utils;
import java.io.IOException;
import java.nio.ByteBuffer;
import java.util.ArrayList;
import java.util.HashMap;
......@@ -34,6 +35,7 @@ import org.apache.iotdb.tsfile.file.metadata.TsDeviceMetadata;
import org.apache.iotdb.tsfile.file.metadata.TsDeviceMetadataIndex;
import org.apache.iotdb.tsfile.file.metadata.TsDeviceMetadataTest;
import org.apache.iotdb.tsfile.file.metadata.TsDigest;
import org.apache.iotdb.tsfile.file.metadata.TsDigest.StatisticType;
import org.apache.iotdb.tsfile.file.metadata.TsFileMetaData;
import org.apache.iotdb.tsfile.file.metadata.TsFileMetaDataTest;
import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
......@@ -116,19 +118,49 @@ public class TestHelper {
return metaData;
}
public static ChunkMetaData createNotCompleteSimpleTimeSeriesChunkMetaData() throws IOException {
ChunkMetaData metaData = new ChunkMetaData(ChunkMetaDataTest.MEASUREMENT_UID,
ChunkMetaDataTest.DATA_TYPE,
ChunkMetaDataTest.FILE_OFFSET, ChunkMetaDataTest.START_TIME, ChunkMetaDataTest.END_TIME
);
metaData.setNumOfPoints(ChunkMetaDataTest.NUM_OF_POINTS);
metaData.setDigest(createNotCompleteSimpleTsDigest());
return metaData;
}
public static MeasurementSchema createSimpleMeasurementSchema() {
return new MeasurementSchema(TimeSeriesMetadataTest.measurementUID,
TSDataType.INT64,
TSEncoding.RLE);
}
public static TsDigest createSimpleTsDigest() {
public static TsDigest createSimpleTsDigest() throws IOException {
TsDigest digest = new TsDigest();
ByteBuffer[] statisticsArray = new ByteBuffer[StatisticType.getTotalTypeNum()];
statisticsArray[StatisticType.min_value.ordinal()] = ByteBuffer
.wrap(BytesUtils.stringToBytes(MIN_VALUE));
statisticsArray[StatisticType.max_value.ordinal()] = ByteBuffer
.wrap(BytesUtils.stringToBytes(MAX_VALUE));
statisticsArray[StatisticType.first_value.ordinal()] = ByteBuffer
.wrap(BytesUtils.stringToBytes(FIRST_VALUE));
statisticsArray[StatisticType.last_value.ordinal()] = ByteBuffer
.wrap(BytesUtils.stringToBytes(LAST_VALUE));
statisticsArray[StatisticType.sum_value.ordinal()] = ByteBuffer
.wrap(BytesUtils.stringToBytes(SUM_VALUE));
digest.setStatistics(statisticsArray);
return digest;
}
public static TsDigest createNotCompleteSimpleTsDigest() throws IOException {
TsDigest digest = new TsDigest();
digest.addStatistics("max", ByteBuffer.wrap(BytesUtils.stringToBytes(MAX_VALUE)));
digest.addStatistics("min", ByteBuffer.wrap(BytesUtils.stringToBytes(MIN_VALUE)));
digest.addStatistics("sum", ByteBuffer.wrap(BytesUtils.stringToBytes(SUM_VALUE)));
digest.addStatistics("first", ByteBuffer.wrap(BytesUtils.stringToBytes(FIRST_VALUE)));
digest.addStatistics("last", ByteBuffer.wrap(BytesUtils.stringToBytes(LAST_VALUE)));
ByteBuffer[] statisticsArray = new ByteBuffer[StatisticType.getTotalTypeNum()];
statisticsArray[StatisticType.first_value.ordinal()] = ByteBuffer
.wrap(BytesUtils.stringToBytes(FIRST_VALUE));
statisticsArray[StatisticType.last_value.ordinal()] = ByteBuffer
.wrap(BytesUtils.stringToBytes(LAST_VALUE));
statisticsArray[StatisticType.sum_value.ordinal()] = ByteBuffer
.wrap(BytesUtils.stringToBytes(SUM_VALUE));
digest.setStatistics(statisticsArray);
return digest;
}
......
tsfile/src/test/java/org/apache/iotdb/tsfile/file/metadata/utils/Utils.java
浏览文件 @ ea6daf13
......@@ -19,10 +19,10 @@
package org.apache.iotdb.tsfile.file.metadata.utils;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertNotNull;
import static org.junit.Assert.assertTrue;
import static org.junit.Assert.fail;
import java.nio.ByteBuffer;
import java.util.List;
import java.util.Map;
import org.apache.iotdb.tsfile.file.header.PageHeader;
......@@ -30,14 +30,16 @@ import org.apache.iotdb.tsfile.file.metadata.ChunkGroupMetaData;
import org.apache.iotdb.tsfile.file.metadata.ChunkMetaData;
import org.apache.iotdb.tsfile.file.metadata.TsDeviceMetadata;
import org.apache.iotdb.tsfile.file.metadata.TsDeviceMetadataIndex;
import org.apache.iotdb.tsfile.file.metadata.TsDigest;
import org.apache.iotdb.tsfile.file.metadata.TsFileMetaData;
import org.apache.iotdb.tsfile.file.metadata.statistics.Statistics;
import org.apache.iotdb.tsfile.write.schema.MeasurementSchema;
import org.junit.Assert;
public class Utils {
private static final double maxError = 0.0001d;
public static void isListEqual(List<?> listA, List<?> listB, String name) {
if ((listA == null) ^ (listB == null)) {
......@@ -70,32 +72,21 @@ public class Utils {
}
}
public static void isMapBufferEqual(Map<String, ByteBuffer> mapA, Map<String, ByteBuffer> mapB,
String name) {
if ((mapA == null) ^ (mapB == null)) {
public static void isTwoTsDigestEqual(TsDigest digestA, TsDigest digestB, String name) {
if ((digestA == null) ^ (digestB == null)) {
System.out.println("error");
fail(String.format("one of %s is null", name));
}
if ((mapA != null) && (mapB != null)) {
if (mapA.size() != mapB.size()) {
fail(String.format("%s size is different", name));
}
for (String key : mapB.keySet()) {
ByteBuffer b = mapB.get(key);
ByteBuffer a = mapA.get(key);
assertTrue(b.equals(a));
}
if (digestA != null) {
Assert.assertEquals(digestA, digestB);
}
}
/**
* when one of A and B is Null, A != B, so test case fails.
*
* @param objectA
* @param objectB
* @param name
* @return false - A and B both are NULL, so we do not need to check whether their members are equal true - A and B
* both are not NULL, so we need to check their members
* @return false - A and B both are NULL, so we do not need to check whether their members are
* equal true - A and B both are not NULL, so we need to check their members
*/
public static boolean isTwoObjectsNotNULL(Object objectA, Object objectB, String name) {
if ((objectA == null) && (objectB == null)) {
......@@ -128,11 +119,9 @@ public class Utils {
assertTrue(metadata1.getNumOfPoints() == metadata2.getNumOfPoints());
assertTrue(metadata1.getStartTime() == metadata2.getStartTime());
assertTrue(metadata1.getEndTime() == metadata2.getEndTime());
if (Utils.isTwoObjectsNotNULL(metadata1.getDigest(), metadata2.getDigest(), "digest")) {
Utils.isMapBufferEqual(metadata1.getDigest().getStatistics(),
metadata2.getDigest().getStatistics(),
"statistics");
}
assertNotNull(metadata1.getDigest());
assertNotNull(metadata2.getDigest());
Utils.isTwoTsDigestEqual(metadata1.getDigest(), metadata2.getDigest(), "TsDigest");
}
}
......@@ -142,9 +131,11 @@ public class Utils {
assertEquals(metadata1.getStartTime(), metadata2.getStartTime());
assertEquals(metadata1.getEndTime(), metadata2.getEndTime());
if (Utils.isTwoObjectsNotNULL(metadata1.getChunkGroupMetaDataList(), metadata2.getChunkGroupMetaDataList(),
if (Utils.isTwoObjectsNotNULL(metadata1.getChunkGroupMetaDataList(),
metadata2.getChunkGroupMetaDataList(),
"Rowgroup metadata list")) {
assertEquals(metadata1.getChunkGroupMetaDataList().size(), metadata2.getChunkGroupMetaDataList().size());
assertEquals(metadata1.getChunkGroupMetaDataList().size(),
metadata2.getChunkGroupMetaDataList().size());
for (int i = 0; i < metadata1.getChunkGroupMetaDataList().size(); i++) {
Utils.isChunkGroupMetaDataEqual(metadata1.getChunkGroupMetaDataList().get(i),
metadata1.getChunkGroupMetaDataList().get(i));
......@@ -219,7 +210,8 @@ public class Utils {
assertTrue(header1.getNumOfValues() == header2.getNumOfValues());
assertTrue(header1.getMaxTimestamp() == header2.getMaxTimestamp());
assertTrue(header1.getMinTimestamp() == header2.getMinTimestamp());
if (Utils.isTwoObjectsNotNULL(header1.getStatistics(), header2.getStatistics(), "statistics")) {
if (Utils
.isTwoObjectsNotNULL(header1.getStatistics(), header2.getStatistics(), "statistics")) {
Utils.isStatisticsEqual(header1.getStatistics(), header2.getStatistics());
}
}
......
tsfile/src/test/java/org/apache/iotdb/tsfile/read/filter/DigestFilterTest.java
浏览文件 @ ea6daf13
......@@ -18,6 +18,7 @@
*/
package org.apache.iotdb.tsfile.read.filter;
import java.nio.ByteBuffer;
import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
import org.apache.iotdb.tsfile.read.filter.basic.Filter;
import org.apache.iotdb.tsfile.read.filter.factory.FilterFactory;
......@@ -31,16 +32,20 @@ public class DigestFilterTest {
BytesUtils.intToBytes(100), TSDataType.INT32);
private DigestForFilter digest2 = new DigestForFilter(101L, 200L, BytesUtils.intToBytes(101),
BytesUtils.intToBytes(200), TSDataType.INT32);
private DigestForFilter digest3 = new DigestForFilter(101L, 200L, (ByteBuffer) null, null,
TSDataType.INT32);
@Test
public void testEq() {
Filter timeEq = TimeFilter.eq(10L);
Assert.assertEquals(true, timeEq.satisfy(digest1));
Assert.assertEquals(false, timeEq.satisfy(digest2));
Assert.assertEquals(false, timeEq.satisfy(digest3));
Filter valueEq = ValueFilter.eq(100);
Assert.assertEquals(true, valueEq.satisfy(digest1));
Assert.assertEquals(false, valueEq.satisfy(digest2));
Assert.assertEquals(true, valueEq.satisfy(digest3));
}
@Test
......@@ -48,10 +53,12 @@ public class DigestFilterTest {
Filter timeGt = TimeFilter.gt(100L);
Assert.assertEquals(false, timeGt.satisfy(digest1));
Assert.assertEquals(true, timeGt.satisfy(digest2));
Assert.assertEquals(true, timeGt.satisfy(digest3));
Filter valueGt = ValueFilter.gt(100);
Assert.assertEquals(false, valueGt.satisfy(digest1));
Assert.assertEquals(true, valueGt.satisfy(digest2));
Assert.assertEquals(true, valueGt.satisfy(digest3));
}
@Test
......@@ -59,10 +66,12 @@ public class DigestFilterTest {
Filter timeGtEq = TimeFilter.gtEq(100L);
Assert.assertEquals(true, timeGtEq.satisfy(digest1));
Assert.assertEquals(true, timeGtEq.satisfy(digest2));
Assert.assertEquals(true, timeGtEq.satisfy(digest3));
Filter valueGtEq = ValueFilter.gtEq(100);
Assert.assertEquals(true, valueGtEq.satisfy(digest1));
Assert.assertEquals(true, valueGtEq.satisfy(digest2));
Assert.assertEquals(true, valueGtEq.satisfy(digest3));
Assert.assertEquals(true, valueGtEq.satisfy(digest3));
}
@Test
......@@ -70,10 +79,12 @@ public class DigestFilterTest {
Filter timeLt = TimeFilter.lt(101L);
Assert.assertEquals(true, timeLt.satisfy(digest1));
Assert.assertEquals(false, timeLt.satisfy(digest2));
Assert.assertEquals(false, timeLt.satisfy(digest3));
Filter valueLt = ValueFilter.lt(101);
Assert.assertEquals(true, valueLt.satisfy(digest1));
Assert.assertEquals(false, valueLt.satisfy(digest2));
Assert.assertEquals(true, valueLt.satisfy(digest3));
}
@Test
......@@ -81,10 +92,12 @@ public class DigestFilterTest {
Filter timeLtEq = TimeFilter.ltEq(101L);
Assert.assertEquals(true, timeLtEq.satisfy(digest1));
Assert.assertEquals(true, timeLtEq.satisfy(digest2));
Assert.assertEquals(true, timeLtEq.satisfy(digest3));
Filter valueLtEq = ValueFilter.ltEq(101);
Assert.assertEquals(true, valueLtEq.satisfy(digest1));
Assert.assertEquals(true, valueLtEq.satisfy(digest2));
Assert.assertEquals(true, valueLtEq.satisfy(digest3));
}
@Test
......@@ -92,10 +105,12 @@ public class DigestFilterTest {
Filter andFilter = FilterFactory.and(TimeFilter.gt(10L), ValueFilter.lt(50));
Assert.assertEquals(true, andFilter.satisfy(digest1));
Assert.assertEquals(false, andFilter.satisfy(digest2));
Assert.assertEquals(true, andFilter.satisfy(digest3));
Filter orFilter = FilterFactory.or(andFilter, TimeFilter.eq(200L));
Assert.assertEquals(true, orFilter.satisfy(digest1));
Assert.assertEquals(true, orFilter.satisfy(digest2));
Assert.assertEquals(true, orFilter.satisfy(digest3));
}
}
Markdown is supported
0% 或 .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑！
想要评论请 注册 或