提交 ba5bc634 编写于 作者: Z Zesong Sun 提交者: Jialin Qiao

Add documents for Shared Storage Architecture (#475)

上级 a92e0d9f
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
# 第10章: 分布式架构
## 存储共享架构
当前,TSFile(包括TSFile文件和相关的数据文件)支持存储在本地文件系统和Hadoop分布式文件系统(HDFS)。配置使用HDFS存储TSFile十分容易。
### 系统架构
当你配置使用HDFS存储TSFile之后,你的数据文件将会被分布式存储。系统架构如下:
<img style="width:100%; max-width:700px; max-height:600px; margin-left:auto; margin-right:auto; display:block;" src="https://user-images.githubusercontent.com/19167280/66922722-35180400-f05a-11e9-8ff0-7dd51716e4a8.png">
### Config and usage
如果你希望将TSFile存储在HDFS上,可以遵循以下步骤:
首先,使用maven打包server和Hadoop模块:`mvn clean package -pl server,hadoop -am -Dmaven.test.skip=true`
然后,将Hadoop模块的target jar包`hadoop-tsfile-0.9.0-SNAPSHOT-jar-with-dependencies.jar`复制到server模块的target lib文件夹 `.../server/target/iotdb-server-0.9.0-SNAPSHOT/lib`下。
编辑`iotdb-engine.properties`中的用户配置。相关配置项包括:
* tsfile\_storage\_fs
|名字| tsfile\_storage\_fs |
|:---:|:---|
|描述| Tsfile和相关数据文件的存储文件系统。目前支持LOCAL(本地文件系统)和HDFS两种|
|类型| String |
|默认值|LOCAL |
|改后生效方式|重启服务器生效|
* hdfs\_ip
|名字| hdfs\_ip |
|:---:|:---|
|描述| 在Tsfile和相关数据文件存储到HDFS的情况下用于配置HDFS的IP|
|类型| String |
|默认值|localhost |
|改后生效方式|重启服务器生效|
* hdfs\_port
|名字| hdfs\_port |
|:---:|:---|
|描述| 在Tsfile和相关数据文件存储到HDFS的情况下用于配置HDFS的端口|
|类型| String |
|默认值|9000 |
|改后生效方式|重启服务器生效|
启动server, Tsfile将会被存储到HDFS上。
如果你想要恢复将TSFile存储到本地文件系统,只需编辑配置项`tsfile_storage_fs``LOCAL`。在这种情况下,如果你已经在HDFS上存储了一些数据文件,你需要将它们下载到本地,并移动到你所配置的数据文件文件夹(默认为`../server/target/iotdb-server-0.9.0-SNAPSHOT/data/data`), 或者重新开始你的整个导入数据过程。
### 常见问题
1. 这个功能支持哪些Hadoop版本?
A: Hadoop 2.x and Hadoop 3.x均可以支持。
2. 当启动服务器或创建时间序列时,我遇到了如下错误:
```
ERROR org.apache.iotdb.tsfile.fileSystem.fsFactory.HDFSFactory:62 - Failed to get Hadoop file system. Please check your dependency of Hadoop module.
```
A: 这表明你没有将Hadoop模块的以来放到IoTDB server中。你可以这样解决:
* 使用Maven打包Hadoop模块: `mvn clean package -pl hadoop -am -Dmaven.test.skip=true`
* 将Hadoop模块的target jar包`hadoop-tsfile-0.9.0-SNAPSHOT-jar-with-dependencies.jar`复制到server模块的target lib文件夹 `.../server/target/iotdb-server-0.9.0-SNAPSHOT/lib`下。
......@@ -63,3 +63,6 @@
* 6-TsFile Spark Connector
* 7-Spark IoTDB Connector
* 8-Watermark Tool
* 9-Log Visualizer
# Chapter 9: Distributed Architecture
* 1-Shared Storage Architecture
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
# Chapter 10: Distributed Architecture
## Shared Storage Architecture
Currently, TSFiles(including both TSFile and related data files) are supported to be stored in local file system and hadoop distributed file system(HDFS). It is very easy to config the storage file system of TSFile.
### System architecture
When you config to store TSFile on HDFS, your data files will be in distributed storage. The system architecture is as below:
<img style="width:100%; max-width:700px; max-height:600px; margin-left:auto; margin-right:auto; display:block;" src="https://user-images.githubusercontent.com/19167280/66922722-35180400-f05a-11e9-8ff0-7dd51716e4a8.png">
### Config and usage
If you want to store TSFile and related data files in HDFS, here are the steps:
First, build server and Hadoop module by: `mvn clean package -pl server,hadoop -am -Dmaven.test.skip=true`
Then, copy the target jar of Hadoop module `hadoop-tsfile-0.9.0-SNAPSHOT-jar-with-dependencies.jar` into server target lib folder `.../server/target/iotdb-server-0.9.0-SNAPSHOT/lib`.
Edit user config in `iotdb-engine.properties`. Related configurations are:
* tsfile\_storage\_fs
|Name| tsfile\_storage\_fs |
|:---:|:---|
|Description| The storage file system of Tsfile and related data files. Currently LOCAL file system and HDFS are supported.|
|Type| String |
|Default|LOCAL |
|Effective|After restart system|
* hdfs\_ip
|Name| hdfs\_ip |
|:---:|:---|
|Description| IP of HDFS if Tsfile and related data files are stored in HDFS|
|Type| String |
|Default|localhost |
|Effective|After restart system|
* hdfs\_port
|Name| hdfs\_port |
|:---:|:---|
|Description| Port of HDFS if Tsfile and related data files are stored in HDFS|
|Type| String |
|Default|9000 |
|Effective|After restart system|
Start server, and Tsfile will be stored on HDFS.
If you'd like to reset storage file system to local, just edit configuration `tsfile_storage_fs` to `LOCAL`. In this situation, if you have already had some data files on HDFS, you should either download them to local and move them to your config data file folder (`../server/target/iotdb-server-0.9.0-SNAPSHOT/data/data` by default), or restart your process and import data to IoTDB.
### Frequent questions
1. What Hadoop version does it support?
A: Both Hadoop 2.x and Hadoop 3.x can be supported.
2. When starting the server or trying to create timeseries, I encounter the error below:
```
ERROR org.apache.iotdb.tsfile.fileSystem.fsFactory.HDFSFactory:62 - Failed to get Hadoop file system. Please check your dependency of Hadoop module.
```
A: It indicates that you forget to put Hadoop module dependency in IoTDB server. You can solve it by:
* Build Hadoop module: `mvn clean package -pl hadoop -am -Dmaven.test.skip=true`
* Copy the target jar of Hadoop module `hadoop-tsfile-0.9.0-SNAPSHOT-jar-with-dependencies.jar` into server target lib folder `.../server/target/iotdb-server-0.9.0-SNAPSHOT/lib`.
Markdown is supported
0% .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑!
想要评论请 注册