Skip to content

I
Iotdb

apache / Iotdb
大约 1 年 前同步成功

通知 25
Star 3344
Fork 916
I
Iotdb

体验新版 GitCode，发现更多精彩内容 >>

未验证 提交 226371c6 编写于 作者: T Tianan Li 提交者： GitHub
浏览文件
操作

Merge pull request #256 from apache/dev_memory_estimate_tool 

[IOTDB-141]Develop memory estimate tool
上级 cbe148e8 629b6dc9
隐藏空白更改
内联 并排
Showing with 852 addition and 119 deletion
docs/Documentation-CHN/UserGuideV0.7.0/8-System Tools/1-Sync.md 0 → 100644
浏览文件 @ 226371c6
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
# 同步工具
<!-- TOC -->
- [同步工具](#同步工具)
- [介绍](#介绍)
- [配置参数](#配置参数)
- [同步工具接收端](#同步工具接收端)
- [同步工具发送端](#同步工具发送端)
- [使用方式](#使用方式)
- [启动同步功能接收端](#启动同步功能接收端)
- [关闭同步功能接收端](#关闭同步功能接收端)
- [启动同步功能发送端](#启动同步功能发送端)
- [关闭同步功能发送端](#关闭同步功能发送端)
<!-- /TOC -->
# 介绍
同步工具是定期将本地磁盘中和新增的已持久化的tsfile文件上传至云端并加载的IoTDB套件工具。
在同步工具的发送端，同步模块是一个独立的进程，独立于本地的IoTDB。通过独立的脚本进行启动和关闭(详见章节`使用方式`)，同步的频率周期可由用户设置。
在同步工具的的接收端，同步模块内嵌于IoTDB的引擎，和IoTDB处于同一个进程中。同步模块监听一个独立的端口，该端口可由用户设置(详见章节`配置参数`)。用户使用前，需要在同步接收端设置同步白名单，以网段形式表示，接收端的同步模块只接受位于白名单网段中的发送端同步的数据.
同步工具具有多对一的发送-接受模式，即一个同步接收端可以同时接受多个同步发送端传输的数据，一个同步发送端只能向一个同步接收端发送数据
> 注意：在使用同步工具前，同步工具的接收端和发送端需要单独配置。
# 配置参数
## 同步工具接收端
同步工具接收端的参数配置位于IoTDB的配置文件iotdb-engine.properties中，其安装目录为$IOTDB_HOME/conf/iotdb-engine.properties。在该配置文件中，有四个参数和同步接收端有关，配置说明如下:
<table>
<tr>
<td colspan="2">参数名: is_sync_enable</td>
</tr>
<tr>
<td width="20%">描述</td>
<td>同步功能开关，配置为true表示接收端允许接收同步的数据并加载，设置为false的时候表示接收端不允许接收同步的数据</td>
</tr>
<tr>
<td>类型</td>
<td>Boolean</td>
</tr>
<tr>
<td>默认值</td>
<td>false</td>
</tr>
<tr>
<td>改后生效方式</td>
<td>重启服务器生效</td>
</tr>
</table>
<table>
<tr>
<td colspan="2">参数名: IP_white_list</td>
</tr>
<tr>
<td width="20%">描述</td>
<td>设置同步功能发送端IP地址的白名单，以网段的形式表示，多个网段之间用逗号分隔。发送端向接收端同步数据时，只有当该发送端IP地址处于该白名单设置的网段范围内，接收端才允许同步操作。如果白名单为空，则接收端不允许任何发送端同步数据。默认接收端接受全部IP的同步请求。</td>
</tr>
<tr>
<td>类型</td>
<td>String</td>
</tr>
<tr>
<td>默认值</td>
<td>0.0.0.0/0</td>
</tr>
<tr>
<td>改后生效方式</td>
<td>重启服务器生效</td>
</tr>
</table>
<table>
<tr>
<td colspan="2">参数名: update_historical_data_possibility</td>
</tr>
<tr>
<td width="20%">描述</td>
<td>同步服务端在合并同步的数据时选择的处理策略。如果同步的数据对历史数据(相比本地该存储组数据的最新时间戳)更新占比超过50%，则建议选择策略1，将参数设置为true，使用该策略对IoTDB系统的写入性能产生较大影响，对机器的CPU占用较小；如果同步的数据对历史数据更新占比少于50%，则建议选择策略2，将参数设置为false，使用该策略对IoTDB系统的写入性能产生较小影响，对机器CPU的占用较大。<br/>
</td>
</tr>
<tr>
<td>类型</td>
<td>Boolean</td>
</tr>
<tr>
<td>默认值</td>
<td>false</td>
</tr>
<tr>
<td>改后生效方式</td>
<td>重启服务器生效</td>
</tr>
</table>
<table>
<tr>
<td colspan="2">参数名: sync_server_port</td>
</tr>
<tr>
<td width="20%">描述</td>
<td>同步接收端服务器监听接口，请确认该端口不是系统保留端口并且未被占用。参数is_sync_enable设置为true时有效，参数is_sync_enable设置为false时无效</td>
</tr>
<tr>
<td>类型</td>
<td>Short Int : [0,65535]</td>
</tr>
<tr>
<td>默认值</td>
<td>5555</td>
</tr>
<tr>
<td>改后生效方式</td>
<td>重启服务器生效</td>
</tr>
</table>
## 同步工具发送端
同步功能发送端的参数配置在一个单独的配置文件中，其安装目录为```$IOTDB_HOME/conf/iotdb-sync-client.properties```。在该配置文件中，有五个参数和同步发送端有关，配置说明如下：
<table>
<tr>
<td colspan="2">参数名: server_ip</td>
</tr>
<tr>
<td width="20%">描述</td>
<td>同步接收端的IP地址</td>
</tr>
<tr>
<td>类型</td>
<td>String</td>
</tr>
<tr>
<td>默认值</td>
<td>127.0.0.1</td>
</tr>
<tr>
<td>改后生效方式</td>
<td>重启同步功能发送端生效</td>
</tr>
</table>
<table>
<tr>
<td colspan="2">参数名: server_port</td>
</tr>
<tr>
<td width="20%">描述</td>
<td>同步接收端服务器监听端口，需要保证该端口和同步接收端配置的监听端口一致</td>
</tr>
<tr>
<td>类型</td>
<td>Short Int : [0,65535]</td>
</tr>
<tr>
<td>默认值</td>
<td>5555</td>
</tr>
<tr>
<td>改后生效方式</td>
<td>重启同步功能发送端生效</td>
</tr>
</table>
<table>
<tr>
<td colspan="2">参数名: sync_period_in_second</td>
</tr>
<tr>
<td width="20%">描述</td>
<td>同步周期，两次同步任务开始时间的间隔，单位为秒(s)</td>
</tr>
<tr>
<td>类型</td>
<td>Int : [0,2147483647]</td>
</tr>
<tr>
<td>默认值</td>
<td>600</td>
</tr>
<tr>
<td>改后生效方式</td>
<td>重启同步功能发送端生效</td>
</tr>
</table>
<table>
<tr>
<td colspan="2">参数名: iotdb_schema_directory</td>
</tr>
<tr>
<td width="20%">描述</td>
<td>同步发送端的IoTDB schema文件的绝对路径，例如$IOTDB_HOME /data/system/schema/mlog.txt(若用户未手动设置schema元数据的路径，则该路径为默认路径)，该参数默认不生效，用户有需求时进行手动设置</td>
</tr>
<tr>
<td>类型</td>
<td>String</td>
</tr>
<tr>
<td>改后生效方式</td>
<td>重启同步功能发送端生效</td>
</tr>
</table>
<table>
<tr>
<td colspan="2">参数名: iotdb_bufferWrite_directory</td>
</tr>
<tr>
<td width="20%">描述</td>
<td>同步发送端的IoTDB 的bufferWrite数据(tsfile文件)目录的绝对路径，定位至bufferWrite目录下，例如： $IOTDB_HOME /data/data/settled(若用户未手动设置数据路径，则该路径为默认路径)，该参数默认不生效，用户有需求时进行手动设置。该参数需要保证和参数iotdb_schema_directory属于同一个IoTDB</td>
</tr>
<tr>
<td>类型</td>
<td>String</td>
</tr>
<tr>
<td>改后生效方式</td>
<td>重启同步功能发送端生效</td>
</tr>
</table>
# 使用方式
## 启动同步功能接收端
1. 配置接收端的参数，例如：
<img style="width:100%; max-width:800px; max-height:600px; margin-left:auto; margin-right:auto; display:block;" src="https://user-images.githubusercontent.com/26211279/59494502-daaa4380-8ebf-11e9-8bce-363e2433005a.png">
2. 启动IoTDB引擎，同步功能接收端会同时启动，启动时LOG日志会出现`IoTDB: start SYNC ServerService successfully`字样，表示同步接收端启动成功，如图所示：
<img style="width:100%; max-width:800px; max-height:600px; margin-left:auto; margin-right:auto; display:block;" src="https://user-images.githubusercontent.com/26211279/59494513-df6ef780-8ebf-11e9-83e1-ee8ae64b76d0.png">
## 关闭同步功能接收端
关闭IoTDB，同步功能接收端会同时关闭。
## 启动同步功能发送端
1. 配置发送端的参数， 如图所示:
<img style="width:100%; max-width:800px; max-height:600px; margin-left:auto; margin-right:auto; display:block;" src="https://user-images.githubusercontent.com/26211279/59494559-f9a8d580-8ebf-11e9-875e-355199c1a1e9.png">
2. 启动同步功能发送端
用户可以使用```$IOTDB_HOME/bin```文件夹下的脚本启动同步功能的发送端
Linux系统与MacOS系统启动命令如下：
```
Shell >$IOTDB_HOME/bin/start-sync-client.sh
```
Windows系统启动命令如下：
```
Shell >$IOTDB_HOME/bin/start-sync-client.bat
```
<img style="width:100%; max-width:800px; max-height:600px; margin-left:auto; margin-right:auto; display:block;" src="https://user-images.githubusercontent.com/26211279/59494951-dc283b80-8ec0-11e9-9575-5d8578c08ceb.png">
## 关闭同步功能发送端
用户可以使用```$IOTDB_HOME/bin```文件夹下的脚本关闭同步功能的发送端。
Linux系统与MacOS系统停止命令如下：
```
Shell >$IOTDB_HOME/bin/stop-sync-client.sh
```
Windows系统停止命令如下：
```
Shell >$IOTDB_HOME/bin/stop-sync-client.bat
```
\ No newline at end of file
docs/Documentation-CHN/UserGuideV0.7.0/8-System Tools/2-Memory Estimation Tool.md 0 → 100644
浏览文件 @ 226371c6
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
# 内存预估工具
<!-- TOC -->
- [内存预估工具](#内存预估工具)
- [介绍](#介绍)
- [输入参数](#输入参数)
- [使用方式](#使用方式)
<!-- /TOC -->
# 介绍
本工具通过用户输入的若干参数，计算出IoTDB运行此负载的最小写内存。(IoTDB中的内存分为三部分：写内存，读内存，预留内存。写内存是用于数据写入分配的内存，三者的比例可在配置文件中设置)，结果以GB为单位。
# 输入参数
本工具使用时，需要输入的参数如下:
<table>
<tr>
<td>参数名</td>
<td>参数说明</td>
<td>示例</td>
<td>是否必需</td>
</tr>
<tr>
<td>-sg | --storagegroup &lt;storage group number&gt;</td>
<td>存储组数量</td>
<td>-sg 20</td>
<td>是</td>
</tr>
<tr>
<td>-ts | --timeseries &lt;total timeseries number&gt;</td>
<td>总时间序列数量</td>
<td>-ts 10000</td>
<td>是</td>
</tr>
<tr>
<td>-mts | --maxtimeseries &lt;max timeseries&gt;</td>
<td>存储组中的最大时间序列的数量，如果时间序列均匀分配在存储组中，本参数可以不设置</td>
<td>-mts 10000</td>
<td>否</td>
</tr>
</table>
在内存预估时，若工具计算需要较长的时间，则会在下方显示出运行进度，便于用户掌握进度。
# 使用方式
用户可以使用```$IOTDB_HOME/bin```文件夹下的脚本使用该工具
Linux系统与MacOS系统启动命令如下：
* 以20个存储组，共10w条时间序列，时间序列在存储组中均分为例：
```
Shell >$IOTDB_HOME/bin/memory-tool.sh calmem -sg 20 -ts 100000
```
* 以20个存储组，共10w条时间序列，存储组中最大时间序列数为50000为例：
```
Shell >$IOTDB_HOME/bin/memory-tool.sh calmem -sg 20 -ts 100000 -tsm -50000
```
Windows系统启动命令如下：
* 以20个存储组，共10w条时间序列，时间序列在存储组中均分为例：
```
Shell >$IOTDB_HOME/bin/memory-tool.bat calmem -sg 20 -ts 100000
```
* 以20个存储组，共10w条时间序列，存储组中最大时间序列数为50000为例：
```
Shell >$IOTDB_HOME/bin/memory-tool.bat calmem -sg 20 -ts 100000 -tsm -50000
```
docs/Documentation/UserGuideV0.7.0/0-Content.md
浏览文件 @ 226371c6
......@@ -52,4 +52,7 @@
# Chapter 7: TsFile
* 1-Installation
* 2-Usage
* 3-Hierarchy
\ No newline at end of file
* 3-Hierarchy
# Chapter 8: System Tools
* 1-Sync.md
* 2-Memory estimation tool.md
docs/Documentation/UserGuideV0.7.0/7-Tools-Sync.md → docs/Documentation/UserGuideV0.7.0/8-System Tools/1-Sync.md
浏览文件 @ 226371c6
......@@ -19,19 +19,23 @@
-->
<!-- TOC -->
# Chapter 7: System Tools
## Data Import
## Outline
<!-- TOC -->
- Introduction
- Configuration
- Sync Receiver
- Sync Sender
- Usage
- Start Sync Receiver
- Stop Sync Receiver
- Start Sync Sender
- Stop Sync Sender
- [Chapter 7: System Tools](#chapter-7-system-tools)
- [Data Import](#data-import)
- [Introduction](#introduction)
- [Configuration](#configuration)
- [Sync Receiver](#sync-receiver)
- [Sync Sender](#sync-sender)
- [Usage](#usage)
- [Start Sync Receiver](#start-sync-receiver)
- [Stop Sync Receiver](#stop-sync-receiver)
- [Start Sync Sender](#start-sync-sender)
- [Stop Sync Sender](#stop-sync-sender)
<!-- /TOC -->
# Introduction
......@@ -54,7 +58,7 @@ The parameter configuration of the sync receiver is located in the configuration
</tr>
<tr>
<td width="30%">Description</td>
<td>Sync function switch, which is configured as true to indicate that the receiver is allowed to receive the data from sender and load it. When set to false, it means that the receiver is not allowed to receive the data from any sender.</td>
<td>Sync function switch, which is configured as true to indicate that the receiver is allowed to receive the data from the sender and load it. When set to false, it means that the receiver is not allowed to receive the data from any sender. </td>
</tr>
<tr>
<td>Type</td>
......@@ -140,7 +144,7 @@ The parameter configuration of the sync receiver is located in the configuration
</table>
## Sync Sender
The parameters of the sync sender are configured in a separate configuration file iotdb-postbackClient.pro-perties with the installation directory of ```$IOTDB_HOME/conf/iotdb-sync-client.properties```. In this configuration file, there are five parameters related to the sync sender. The configuration instructions are as follows:
The parameters of the sync sender are configured in a separate configuration file iotdb-sync-client.properties with the installation directory of ```$IOTDB_HOME/conf/iotdb-sync-client.properties```. In this configuration file, there are five parameters related to the sync sender. The configuration instructions are as follows:
<table>
<tr>
<td colspan="2">parameter: server_ip</td>
......@@ -257,7 +261,7 @@ Stop IoTDB and the sync receiver will be closed at the same time.
1. Set up parameters of sync sender. For example:
<img style="width:100%; max-width:800px; max-height:600px; margin-left:auto; margin-right:auto; display:block;" src="https://user-images.githubusercontent.com/26211279/59494559-f9a8d580-8ebf-11e9-875e-355199c1a1e9.png">
2. Start sync sender
Users can use the scripts under the $IOTDB_HOME/bin folder to start the sync sender.
Users can use the scripts under the ```$IOTDB_HOME/bin``` folder to start the sync sender.
For Linux and Mac OS X users:
```
Shell >$IOTDB_HOME/bin/start-sync-client.sh
......@@ -269,7 +273,7 @@ For Windows users:
<img style="width:100%; max-width:800px; max-height:600px; margin-left:auto; margin-right:auto; display:block;" src="https://user-images.githubusercontent.com/26211279/59494951-dc283b80-8ec0-11e9-9575-5d8578c08ceb.png">
## Stop Sync Sender
Users can use the scripts under the $IOTDB_HOME/bin folder to stop the sync sender.
Users can use the scripts under the ```$IOTDB_HOME/bin``` folder to stop the sync sender.
For Linux and Mac OS X users:
```
Shell >$IOTDB_HOME/bin/stop-sync-client.sh
......
docs/Documentation/UserGuideV0.7.0/8-System Tools/2-Memory Estimation Tool.md 0 → 100644
浏览文件 @ 226371c6
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
# Memory Estimation Tool
# Introduction
This tool calculates the minimum memory for writing to meet specific workload through a number of parameters input by users. (Memory in IoTDB is divided into three parts: write memory, read memory and reserve memory. Write memory is used for data write allocation. The ratio of the three can be set in the configuration file) The unit of result is in GB.
# Input parameters
When using this tool, the parameters needed to be input are as follows:
<table>
<tr>
<td>Parameter</td>
<td>Parameter Description</td>
<td>Example</td>
<td>Necessary</td>
</tr>
<tr>
<td>-sg | --storagegroup &lt;storage group number&gt;</td>
<td>storage group number</td>
<td>-sg 20</td>
<td>true</td>
</tr>
<tr>
<td>-ts | --timeseries &lt;total timeseries number&gt;</td>
<td>total timeseries number</td>
<td>-ts 10000</td>
<td>true</td>
</tr>
<tr>
<td>-mts | --maxtimeseries &lt;max timeseries&gt;</td>
<td>maximum number of timeseries among storage groups.If the time series are evenly distributed in the storage group, this parameter may not be set.</td>
<td>-mts 10000</td>
<td>false</td>
</tr>
</table>
In memory estimation, if the calculation takes a long time, the tool will show the running progress below, which is convenient for users to master the progress.
# Usage
Users can use the tool using scripts under the ``IOTDB_HOME/bin`folder.
For Linux and Mac OS X users:
* Assume that there are 20 storage groups, 10w timeseries and timeseries are evenly distributed in the storage groups:
```
Shell >$IOTDB_HOME/bin/memory-tool.sh calmem -sg 20 -ts 100000
```
* Assume that there are 20 storage groups, 10w timeseries and maximum timeseries number among storage groups is 50000:
```
Shell >$IOTDB_HOME/bin/memory-tool.sh calmem -sg 20 -ts 100000 -tsm -50000
```
For Windows users:
* Assume that there are 20 storage groups, 10w timeseries and timeseries are evenly distributed in the storage groups:
```
Shell >$IOTDB_HOME/bin/memory-tool.bat calmem -sg 20 -ts 100000
```
* Assume that there are 20 storage groups, 10w timeseries and maximum timeseries number among storage groups is 50000:
```
Shell >$IOTDB_HOME/bin/memory-tool.bat calmem -sg 20 -ts 100000 -tsm -50000
```
iotdb/iotdb/bin/memory-tool.bat 0 → 100755
浏览文件 @ 226371c6
@REM
@REM Licensed to the Apache Software Foundation (ASF) under one
@REM or more contributor license agreements. See the NOTICE file
@REM distributed with this work for additional information
@REM regarding copyright ownership. The ASF licenses this file
@REM to you under the Apache License, Version 2.0 (the
@REM "License"); you may not use this file except in compliance
@REM with the License. You may obtain a copy of the License at
@REM
@REM http://www.apache.org/licenses/LICENSE-2.0
@REM
@REM Unless required by applicable law or agreed to in writing,
@REM software distributed under the License is distributed on an
@REM "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
@REM KIND, either express or implied. See the License for the
@REM specific language governing permissions and limitations
@REM under the License.
@REM
@echo off
if "%OS%" == "Windows_NT" setlocal
pushd %~dp0..
if NOT DEFINED IOTDB_HOME set IOTDB_HOME=%CD%
popd
set IOTDB_CONF=%IOTDB_HOME%\conf
if NOT DEFINED MAIN_CLASS set MAIN_CLASS=org.apache.iotdb.db.tools.MemEst.MemEstTool
if NOT DEFINED JAVA_HOME goto :err
@REM -----------------------------------------------------------------------------
@REM JVM Opts we'll use in legacy run or installation
set JAVA_OPTS=-ea^
-Dlogback.configurationFile="%IOTDB_CONF%\logback-tool.xml"^
-DIOTDB_HOME=%IOTDB_HOME%
@REM ***** CLASSPATH library setting *****
@REM Ensure that any user defined CLASSPATH variables are not used on startup
set CLASSPATH="%IOTDB_HOME%\lib"
@REM For each jar in the IOTDB_HOME lib directory call append to build the CLASSPATH variable.
for %%i in ("%IOTDB_HOME%\lib\*.jar") do call :append "%%i"
goto okClasspath
:append
set CLASSPATH=%CLASSPATH%;%1
goto :eof
@REM -----------------------------------------------------------------------------
:okClasspath
"%JAVA_HOME%\bin\java" %JAVA_OPTS% %JAVA_OPTS% -cp "%CLASSPATH%" %MAIN_CLASS% %*
goto finally
:err
echo JAVA_HOME environment variable must be set!
pause
@REM -----------------------------------------------------------------------------
:finally
ENDLOCAL
\ No newline at end of file
iotdb/iotdb/bin/memory-tool.sh 0 → 100755
浏览文件 @ 226371c6
#!/bin/sh
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
#
if [ -z "${IOTDB_HOME}" ]; then
export IOTDB_HOME="$(cd "`dirname "$0"`"/..; pwd)"
fi
IOTDB_CONF=${IOTDB_HOME}/conf
CLASSPATH=""
for f in ${IOTDB_HOME}/lib/*.jar; do
CLASSPATH=${CLASSPATH}":"$f
done
MAIN_CLASS=org.apache.iotdb.db.tools.MemEst.MemEstTool
if [ -n "$JAVA_HOME" ]; then
for java in "$JAVA_HOME"/bin/amd64/java "$JAVA_HOME"/bin/java; do
if [ -x "$java" ]; then
JAVA="$java"
break
fi
done
else
JAVA=java
fi
iotdb_parms="-Dlogback.configurationFile=${IOTDB_CONF}/logback-tool.xml"
exec "$JAVA" $iotdb_parms -cp "$CLASSPATH" "$MAIN_CLASS" "$@"
\ No newline at end of file
iotdb/iotdb/bin/start-server.bat
浏览文件 @ 226371c6
......@@ -78,7 +78,7 @@ set JAVA_OPTS=-ea^
@REM Ensure that any user defined CLASSPATH variables are not used on startup
set CLASSPATH="%IOTDB_HOME%\lib"
REM For each jar in the IOTDB_HOME lib directory call append to build the CLASSPATH variable.
@REM For each jar in the IOTDB_HOME lib directory call append to build the CLASSPATH variable.
for %%i in ("%IOTDB_HOME%\lib\*.jar") do call :append "%%i"
set CLASSPATH=%CLASSPATH%;iotdb.IoTDB
goto okClasspath
......@@ -87,7 +87,7 @@ goto okClasspath
set CLASSPATH=%CLASSPATH%;%1
goto :eof
REM -----------------------------------------------------------------------------
@REM -----------------------------------------------------------------------------
:okClasspath
rem echo CLASSPATH: %CLASSPATH%
......
iotdb/iotdb/bin/start-sync-client.bat
浏览文件 @ 226371c6
......@@ -44,7 +44,7 @@ set JAVA_OPTS=-ea^
@REM Ensure that any user defined CLASSPATH variables are not used on startup
set CLASSPATH="%IOTDB_HOME%\lib"
REM For each jar in the IOTDB_HOME lib directory call append to build the CLASSPATH variable.
@REM For each jar in the IOTDB_HOME lib directory call append to build the CLASSPATH variable.
for %%i in ("%IOTDB_HOME%\lib\*.jar") do call :append "%%i"
set CLASSPATH=%CLASSPATH%;SyncClient
goto okClasspath
......@@ -53,10 +53,10 @@ goto okClasspath
set CLASSPATH=%CLASSPATH%;%1
goto :eof
REM -----------------------------------------------------------------------------
@REM -----------------------------------------------------------------------------
:okClasspath
rem echo CLASSPATH: %CLASSPATH%
@rem echo CLASSPATH: %CLASSPATH%
"%JAVA_HOME%\bin\java" %JAVA_OPTS% %IOTDB_HEAP_OPTS% -cp %CLASSPATH% %IOTDB_JMX_OPTS% %MAIN_CLASS%
goto finally
......
iotdb/iotdb/conf/logback-tool.xml 0 → 100644
浏览文件 @ 226371c6
<?xml version="1.0" encoding="UTF-8"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<configuration/>
iotdb/pom.xml
浏览文件 @ 226371c6
......@@ -70,6 +70,10 @@
<artifactId>commons-lang3</artifactId>
<version>${common.lang3.version}</version>
</dependency>
<dependency>
<groupId>io.airlift</groupId>
<artifactId>airline</artifactId>
</dependency>
<!-- for mocked test-->
<dependency>
<groupId>org.powermock</groupId>
......
iotdb/src/main/java/org/apache/iotdb/db/auth/user/LocalFileUserAccessor.java
浏览文件 @ 226371c6
......@@ -26,7 +26,6 @@ import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.nio.ByteBuffer;
import java.nio.file.Files;
import java.util.ArrayList;
import java.util.HashSet;
import java.util.List;
......
iotdb/src/main/java/org/apache/iotdb/db/conf/IoTDBConfig.java
浏览文件 @ 226371c6
......@@ -213,7 +213,7 @@ public class IoTDBConfig {
/**
* The memory used for stat performance.
*/
private int performance_stat_memory_in_kb = 20;
private int performanceStatMemoryInKB = 20;
/**
* whether use chunkBufferPool.
*/
......@@ -575,12 +575,12 @@ public class IoTDBConfig {
this.performanceStatDisplayInterval = performanceStatDisplayInterval;
}
public int getPerformance_stat_memory_in_kb() {
return performance_stat_memory_in_kb;
public int getPerformanceStatMemoryInKB() {
return performanceStatMemoryInKB;
}
public void setPerformance_stat_memory_in_kb(int performance_stat_memory_in_kb) {
this.performance_stat_memory_in_kb = performance_stat_memory_in_kb;
public void setPerformanceStatMemoryInKB(int performanceStatMemoryInKB) {
this.performanceStatMemoryInKB = performanceStatMemoryInKB;
}
public long getMemtableSizeThreshold() {
......
iotdb/src/main/java/org/apache/iotdb/db/conf/IoTDBDescriptor.java
浏览文件 @ 226371c6
......@@ -226,9 +226,9 @@ public class IoTDBDescriptor {
conf.setPerformanceStatDisplayInterval(Long
.parseLong(properties.getProperty("performance_stat_display_interval",
Long.toString(conf.getPerformanceStatDisplayInterval())).trim()));
conf.setPerformance_stat_memory_in_kb(Integer
conf.setPerformanceStatMemoryInKB(Integer
.parseInt(properties.getProperty("performance_stat_memory_in_kb",
Integer.toString(conf.getPerformance_stat_memory_in_kb())).trim()));
Integer.toString(conf.getPerformanceStatMemoryInKB())).trim()));
} catch (IOException e) {
logger.warn("Cannot load config file because, use default configuration", e);
} catch (Exception e) {
......
iotdb/src/main/java/org/apache/iotdb/db/conf/adapter/CompressionRatio.java
浏览文件 @ 226371c6
......@@ -70,11 +70,7 @@ public class CompressionRatio {
private CompressionRatio() {
directory = new File(
FilePathUtils.regularizePath(CONFIG.getSystemDir()) + COMPRESSION_RATIO_DIR);
try {
restore();
} catch (IOException e) {
LOGGER.error("Can not restore CompressionRatio", e);
}
restore();
}
/**
......@@ -125,8 +121,10 @@ public class CompressionRatio {
/**
* Restore compression ratio statistics from disk when system restart
*/
void restore() throws IOException {
checkDirectoryExist();
void restore() {
if (!directory.exists()) {
return;
}
File[] ratioFiles = directory.listFiles((dir, name) -> name.startsWith(FILE_PREFIX));
if (ratioFiles != null && ratioFiles.length > 0) {
long maxTimes = 0;
......
iotdb/src/main/java/org/apache/iotdb/db/conf/adapter/IoTDBConfigDynamicAdapter.java
浏览文件 @ 226371c6
......@@ -20,7 +20,6 @@ package org.apache.iotdb.db.conf.adapter;
import org.apache.iotdb.db.conf.IoTDBConfig;
import org.apache.iotdb.db.conf.IoTDBDescriptor;
import org.apache.iotdb.db.engine.StorageEngine;
import org.apache.iotdb.db.exception.ConfigAdjusterException;
import org.apache.iotdb.db.metadata.MManager;
import org.apache.iotdb.db.rescon.PrimitiveArrayPool;
......@@ -37,10 +36,11 @@ import org.slf4j.LoggerFactory;
*
* 1. maxMemTableNum. This parameter represents the size of the MemTable available in the MemTable
* pool, which is closely related to the number of storage groups. When adding or deleting a storage
* group, the parameter also adds or deletes two MemTables. The reason why adding or deleting two
* group, the parameter also adds or deletes four MemTables. The reason why adding or deleting four
* MemTables is that when the system is running stably, the speed of the flush operation is faster
* than that of data writing, so one is used for the Flush process and the other is used for data
* writing. Otherwise, the system should limit the speed of data writing to maintain stability.
* writing. Otherwise, the system should limit the speed of data writing to maintain stability. And
* two for sequence data, two for unsequence data.
*
* 2. memtableSize. This parameter determines the threshold value for the MemTable in memory to be
* flushed into disk. When the system load increases, the parameter should be set smaller so that
......@@ -75,16 +75,10 @@ public class IoTDBConfigDynamicAdapter implements IDynamicAdapter {
// static parameter section
/**
* When the size of the adjusted MemTable decreases more than this parameter, trigger the global
* flush operation and flush all MemTable that meets the flush condition to disk.
*/
private static final double FLUSH_THRESHOLD = 0.2;
/**
* Maximum amount of memory allocated for write process.
*/
private static final long ALLOCATE_MEMORY_FOR_WRITE = CONFIG.getAllocateMemoryForWrite();
private static long allocateMemoryForWrite = CONFIG.getAllocateMemoryForWrite();
/**
* Metadata size of per timeseries, the default value is 2KB.
......@@ -128,26 +122,31 @@ public class IoTDBConfigDynamicAdapter implements IDynamicAdapter {
@Override
public synchronized boolean tryToAdaptParameters() {
boolean canAdjust = true;
long memtableSizeInByte = calcMemTableSize();
double ratio = CompressionRatio.getInstance().getRatio();
long memtableSizeInByte = calcMemTableSize(ratio);
long memTableSizeFloorThreshold = getMemTableSizeFloorThreshold();
long tsFileSize = CONFIG.getTsFileSizeThreshold();
long tsFileSizeThreshold = CONFIG.getTsFileSizeThreshold();
if (memtableSizeInByte < memTableSizeFloorThreshold) {
LOGGER.debug("memtableSizeInByte {} is smaller than memTableSizeFloorThreshold {}",
memtableSizeInByte, memTableSizeFloorThreshold);
tsFileSize = calcTsFileSize(memTableSizeFloorThreshold);
memtableSizeInByte = memTableSizeFloorThreshold + ((tsFileSize - memTableSizeFloorThreshold) >> 1);
if (tsFileSize < memTableSizeFloorThreshold) {
tsFileSizeThreshold = calcTsFileSizeThreshold(memTableSizeFloorThreshold);
if ((long) (tsFileSizeThreshold * ratio) < memTableSizeFloorThreshold) {
canAdjust = false;
} else {
// memtableSizeInByte need to be larger than memTableSizeFloorThreshold
memtableSizeInByte = Math.max(memTableSizeFloorThreshold,
memTableSizeFloorThreshold + (((long) (tsFileSizeThreshold * ratio) - memTableSizeFloorThreshold)
>> 1));
}
}
if (canAdjust) {
CONFIG.setMaxMemtableNumber(maxMemTableNum);
CONFIG.setTsFileSizeThreshold(tsFileSize);
CONFIG.setTsFileSizeThreshold(tsFileSizeThreshold);
CONFIG.setMemtableSizeThreshold(memtableSizeInByte);
LOGGER.debug(
"After adjusting, max memTable num is {}, tsFile threshold is {}, memtableSize is {}, memTableSizeFloorThreshold is {}",
maxMemTableNum, tsFileSize, memtableSizeInByte, memTableSizeFloorThreshold);
maxMemTableNum, tsFileSizeThreshold, memtableSizeInByte, memTableSizeFloorThreshold);
currentMemTableSize = memtableSizeInByte;
}
if (!initialized) {
......@@ -163,12 +162,11 @@ public class IoTDBConfigDynamicAdapter implements IDynamicAdapter {
*
* @return MemTable byte size. If the value is -1, there is no valid solution.
*/
private int calcMemTableSize() {
double ratio = CompressionRatio.getInstance().getRatio();
private int calcMemTableSize(double ratio) {
// when unit is byte, it's likely to cause Long type overflow.
// so when b is larger than Integer.MAC_VALUE use the unit KB.
double a = ratio * maxMemTableNum;
double b = (ALLOCATE_MEMORY_FOR_WRITE - staticMemory) * ratio;
double b = (allocateMemoryForWrite - staticMemory) * ratio;
int magnification = b > Integer.MAX_VALUE ? 1024 : 1;
b /= magnification;
double c = (double) CONFIG.getTsFileSizeThreshold() * maxMemTableNum * CHUNK_METADATA_SIZE_IN_BYTE
......@@ -186,8 +184,8 @@ public class IoTDBConfigDynamicAdapter implements IDynamicAdapter {
* @param memTableSize MemTable size
* @return Tsfile byte threshold
*/
private long calcTsFileSize(long memTableSize) {
return (long) ((ALLOCATE_MEMORY_FOR_WRITE - maxMemTableNum * memTableSize - staticMemory) * CompressionRatio
private long calcTsFileSizeThreshold(long memTableSize) {
return (long) ((allocateMemoryForWrite - maxMemTableNum * memTableSize - staticMemory) * CompressionRatio
.getInstance().getRatio()
* memTableSize / (maxMemTableNum * CHUNK_METADATA_SIZE_IN_BYTE * MManager.getInstance()
.getMaximalSeriesNumberAmongStorageGroups()));
......@@ -198,7 +196,7 @@ public class IoTDBConfigDynamicAdapter implements IDynamicAdapter {
* occupied by each value is 8 bytes. The reason for multiplying 2 is that the timestamp also
* takes 8 bytes.
*/
private int getMemTableSizeFloorThreshold() {
private long getMemTableSizeFloorThreshold() {
return MManager.getInstance().getMaximalSeriesNumberAmongStorageGroups()
* PrimitiveArrayPool.ARRAY_SIZE * Long.BYTES * 2;
}
......@@ -253,6 +251,7 @@ public class IoTDBConfigDynamicAdapter implements IDynamicAdapter {
totalTimeseries = 0;
staticMemory = 0;
maxMemTableNum = MEM_TABLE_AVERAGE_QUEUE_LEN;
allocateMemoryForWrite = CONFIG.getAllocateMemoryForWrite();
initialized = false;
}
......
iotdb/src/main/java/org/apache/iotdb/db/cost/statistic/Measurement.java
浏览文件 @ 226371c6
......@@ -120,7 +120,7 @@ public class Measurement implements MeasurementMBean, IService {
IoTDBConfig tdbConfig = IoTDBDescriptor.getInstance().getConfig();
isEnableStat = tdbConfig.isEnablePerformanceStat();
displayIntervalInMs = tdbConfig.getPerformanceStatDisplayInterval();
int memoryInKb = tdbConfig.getPerformance_stat_memory_in_kb();
int memoryInKb = tdbConfig.getPerformanceStatMemoryInKB();
queueSize = memoryInKb * 1000 / Operation.values().length / 8;
operationLatenciesQueue = new ConcurrentCircularArray[Operation.values().length];
......
iotdb/src/main/java/org/apache/iotdb/db/engine/memtable/AbstractMemTable.java
浏览文件 @ 226371c6
......@@ -180,54 +180,6 @@ public abstract class AbstractMemTable implements IMemTable {
this.modifications.add(deletion);
}
/**
* If chunk contains data with timestamp less than 'timestamp', create a copy and delete all those
* data. Otherwise return null.
*
* @param chunk the source chunk.
* @param timestamp the upper-bound of deletion time.
* @return A reduced copy of chunk if chunk contains data with timestamp less than 'timestamp', of
* null.
*/
private IWritableMemChunk filterChunk(IWritableMemChunk chunk, long timestamp) {
if (!chunk.isEmpty() && chunk.getMinTime() <= timestamp) {
//TODO we can avoid sorting data here by scanning data once.
List<TimeValuePair> timeValuePairs = chunk.getSortedTimeValuePairList();
TSDataType dataType = chunk.getType();
IWritableMemChunk newChunk = genMemSeries(dataType);
for (TimeValuePair pair : timeValuePairs) {
if (pair.getTimestamp() > timestamp) {
switch (dataType) {
case BOOLEAN:
newChunk.putBoolean(pair.getTimestamp(), pair.getValue().getBoolean());
break;
case DOUBLE:
newChunk.putDouble(pair.getTimestamp(), pair.getValue().getDouble());
break;
case INT64:
newChunk.putLong(pair.getTimestamp(), pair.getValue().getLong());
break;
case INT32:
newChunk.putInt(pair.getTimestamp(), pair.getValue().getInt());
break;
case FLOAT:
newChunk.putFloat(pair.getTimestamp(), pair.getValue().getFloat());
break;
case TEXT:
newChunk.putBinary(pair.getTimestamp(), pair.getValue().getBinary());
break;
default:
throw new UnsupportedOperationException("Unknown datatype: " + dataType);
}
}
}
TVListAllocator.getInstance().release(dataType, chunk.getTVList());
return newChunk;
}
return null;
}
public void setVersion(long version) {
this.version = version;
}
......
iotdb/src/main/java/org/apache/iotdb/db/engine/memtable/ChunkBufferPool.java
浏览文件 @ 226371c6
......@@ -37,7 +37,7 @@ public class ChunkBufferPool {
private static final Deque<ChunkBuffer> availableChunkBuffer = new ArrayDeque<>();
private int size = 0;
private long size = 0;
private static final int WAIT_TIME = 2000;
......@@ -52,7 +52,7 @@ public class ChunkBufferPool {
synchronized (availableChunkBuffer) {
//we use the memtable number * maximal series number in one StroageGroup * 2 as the capacity
int capacity =
long capacity =
2 * MManager.getInstance().getMaximalSeriesNumberAmongStorageGroups() * IoTDBDescriptor
.getInstance().getConfig().getMaxMemtableNumber() + 100000;
if (availableChunkBuffer.isEmpty() && size < capacity) {
......@@ -91,7 +91,7 @@ public class ChunkBufferPool {
synchronized (availableChunkBuffer) {
chunkBuffer.reset();
//we use the memtable number * maximal series number in one StroageGroup as the capacity
int capacity =
long capacity =
MManager.getInstance().getMaximalSeriesNumberAmongStorageGroups() * IoTDBDescriptor
.getInstance().getConfig().getMaxMemtableNumber();
if (size > capacity) {
......
iotdb/src/main/java/org/apache/iotdb/db/metadata/MManager.java
浏览文件 @ 226371c6
......@@ -74,7 +74,7 @@ public class MManager {
private RandomDeleteCache<String, MNode> mNodeCache;
private Map<String, Integer> seriesNumberInStorageGroups = new HashMap<>();
private int maxSeriesNumberAmongStorageGroup;
private long maxSeriesNumberAmongStorageGroup;
private boolean initialized;
private MManager() {
......@@ -1196,11 +1196,11 @@ public class MManager {
/**
* Only for test
*/
public void setMaxSeriesNumberAmongStorageGroup(int maxSeriesNumberAmongStorageGroup) {
public void setMaxSeriesNumberAmongStorageGroup(long maxSeriesNumberAmongStorageGroup) {
this.maxSeriesNumberAmongStorageGroup = maxSeriesNumberAmongStorageGroup;
}
public int getMaximalSeriesNumberAmongStorageGroups() {
public long getMaximalSeriesNumberAmongStorageGroups() {
return maxSeriesNumberAmongStorageGroup;
}
......
iotdb/src/main/java/org/apache/iotdb/db/tools/MemEst/MemEstTool.java 0 → 100644
浏览文件 @ 226371c6
/**
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
package org.apache.iotdb.db.tools.MemEst;
import com.google.common.base.Throwables;
import com.google.common.collect.Lists;
import io.airlift.airline.Cli;
import io.airlift.airline.Help;
import io.airlift.airline.ParseArgumentsMissingException;
import io.airlift.airline.ParseArgumentsUnexpectedException;
import io.airlift.airline.ParseCommandMissingException;
import io.airlift.airline.ParseCommandUnrecognizedException;
import io.airlift.airline.ParseOptionConversionException;
import io.airlift.airline.ParseOptionMissingException;
import io.airlift.airline.ParseOptionMissingValueException;
import java.io.File;
import java.io.IOException;
import java.util.List;
import org.apache.commons.io.FileUtils;
import org.apache.iotdb.db.conf.IoTDBDescriptor;
public class MemEstTool {
public static void main(String... args) throws IOException {
List<Class<? extends Runnable>> commands = Lists.newArrayList(
Help.class,
MemEstToolCmd.class
);
Cli.CliBuilder<Runnable> builder = Cli.builder("memory-tool");
builder.withDescription("Estimate memory for writing")
.withDefaultCommand(Help.class)
.withCommands(commands);
Cli<Runnable> parser = builder.build();
int status = 0;
try {
Runnable parse = parser.parse(args);
parse.run();
} catch (IllegalArgumentException |
IllegalStateException |
ParseArgumentsMissingException |
ParseArgumentsUnexpectedException |
ParseOptionConversionException |
ParseOptionMissingException |
ParseOptionMissingValueException |
ParseCommandMissingException |
ParseCommandUnrecognizedException e) {
badUse(e);
status = 1;
} catch (Exception e) {
err(Throwables.getRootCause(e));
status = 2;
}
FileUtils.deleteDirectory(new File(IoTDBDescriptor.getInstance().getConfig().getBaseDir()));
System.exit(status);
}
private static void badUse(Exception e) {
System.out.println("memory-tool: " + e.getMessage());
System.out.println("See 'memory-tool help' or 'memory-tool help <command>'.");
}
private static void err(Throwable e) {
System.err.println("error: " + e.getMessage());
System.err.println("-- StackTrace --");
System.err.println(Throwables.getStackTraceAsString(e));
}
}
iotdb/src/main/java/org/apache/iotdb/db/tools/MemEst/MemEstToolCmd.java 0 → 100644
浏览文件 @ 226371c6
/**
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
package org.apache.iotdb.db.tools.MemEst;
import io.airlift.airline.Command;
import io.airlift.airline.Option;
import io.airlift.airline.OptionType;
import org.apache.iotdb.db.conf.IoTDBConfig;
import org.apache.iotdb.db.conf.IoTDBConstant;
import org.apache.iotdb.db.conf.IoTDBDescriptor;
import org.apache.iotdb.db.conf.adapter.IoTDBConfigDynamicAdapter;
import org.apache.iotdb.db.exception.ConfigAdjusterException;
import org.apache.iotdb.db.metadata.MManager;
@Command(name = "calmem", description = "calculate minimum memory required for writing based on the number of storage groups and timeseries")
public class MemEstToolCmd implements Runnable {
@Option(title = "storage group number", name = {"-sg",
"--storagegroup"}, description = "Storage group number")
private String sgNumString = "10";
@Option(title = "total timeseries number", name = {"-ts",
"--timeseries"}, description = "Total timeseries number")
private String tsNumString = "1000";
@Option(title = "max timeseries", name = {"-mts",
"--maxtimeseries"}, description = "Maximum timeseries number among storage groups, make sure that it's smaller than total timeseries number")
private String maxTsNumString = "0";
@Override
public void run() {
IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
long memTableSize = config.getMemtableSizeThreshold();
int maxMemtableNumber = config.getMaxMemtableNumber();
long tsFileSize = config.getTsFileSizeThreshold();
long memory = IoTDBConstant.GB;
long sgNum = Long.parseLong(sgNumString);
long tsNum = Long.parseLong(tsNumString);
long maxTsNum = Long.parseLong(maxTsNumString);
long maxTsNumValid = maxTsNum;
while (true) {
// init parameter
config.setAllocateMemoryForWrite(memory);
config.setMemtableSizeThreshold(memTableSize);
config.setMaxMemtableNumber(maxMemtableNumber);
config.setTsFileSizeThreshold(tsFileSize);
IoTDBConfigDynamicAdapter.getInstance().reset();
IoTDBConfigDynamicAdapter.getInstance().setInitialized(true);
MManager.getInstance().clear();
int sgCnt = 1;
int tsCnt = 1;
try {
for (; sgCnt <= sgNum; sgCnt++) {
IoTDBConfigDynamicAdapter.getInstance().addOrDeleteStorageGroup(1);
}
for (; tsCnt <= tsNum; tsCnt++) {
IoTDBConfigDynamicAdapter.getInstance().addOrDeleteTimeSeries(1);
if(maxTsNum == 0){
maxTsNumValid = tsCnt / sgNum + 1;
} else {
maxTsNumValid = Math.min(tsCnt, maxTsNum);
maxTsNumValid = Math.max(maxTsNumValid, tsCnt / sgNum + 1);
}
MManager.getInstance().setMaxSeriesNumberAmongStorageGroup(maxTsNumValid);
}
} catch (ConfigAdjusterException e) {
if(sgCnt > sgNum) {
System.out
.print(String.format("Memory estimation progress : %d%%\r", tsCnt * 100 / tsNum));
}
memory += IoTDBConstant.GB;
continue;
}
break;
}
System.out.println(String
.format("Memory for writing: %dGB, SG: %d, TS: %d, MTS: %d", memory / IoTDBConstant.GB,
sgNum, tsNum, maxTsNumValid));
}
}
iotdb/src/main/java/org/apache/iotdb/db/tools/WalChecker.java
浏览文件 @ 226371c6
......@@ -16,7 +16,6 @@
* specific language governing permissions and limitations
* under the License.
*/
package org.apache.iotdb.db.tools;
import static org.apache.iotdb.db.writelog.node.ExclusiveWriteLogNode.WAL_FILE_NAME;
......
iotdb/src/main/java/org/apache/iotdb/db/writelog/recover/TsFileRecoverPerformer.java
浏览文件 @ 226371c6
......@@ -25,14 +25,11 @@ import java.io.File;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.Map.Entry;
import java.util.concurrent.ExecutionException;
import java.util.stream.Collectors;
import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
import org.apache.iotdb.db.engine.memtable.IMemTable;
import org.apache.iotdb.db.engine.memtable.MemTableFlushTask;
import org.apache.iotdb.db.engine.memtable.PrimitiveMemTable;
import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
import org.apache.iotdb.db.engine.version.VersionController;
import org.apache.iotdb.db.exception.ProcessorException;
import org.apache.iotdb.db.writelog.manager.MultiFileLogNodeManager;
......
iotdb/src/test/java/org/apache/iotdb/db/conf/adapter/IoTDBConfigDynamicAdapterTest.java
浏览文件 @ 226371c6
......@@ -131,6 +131,7 @@ public class IoTDBConfigDynamicAdapterTest {
MManager.getInstance().setMaxSeriesNumberAmongStorageGroup(i / 30 + 1);
}
} catch (ConfigAdjusterException e) {
System.out.println(i);
assertEquals("The IoTDB system load is too large to add timeseries.", e.getMessage());
}
int j =0;
......
pom.xml
浏览文件 @ 226371c6
......@@ -59,6 +59,7 @@
<common.io.version>2.5</common.io.version>
<commons.collections4>4.0</commons.collections4>
<thrift.version>0.9.3</thrift.version>
<airline.version>0.8</airline.version>
<!-- URL of the ASF SonarQube server -->
<sonar.host.url>https://builds.apache.org/analysis</sonar.host.url>
<!-- Exclude all generated code -->
......@@ -95,6 +96,11 @@
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>io.airlift</groupId>
<artifactId>airline</artifactId>
<version>${airline.version}</version>
</dependency>
</dependencies>
</dependencyManagement>
<dependencies>
......
Markdown is supported
0% 或 .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑！
想要评论请 注册 或