未验证 提交 157be8f2 编写于 作者: B Boris 提交者: GitHub

[IOTDB-937] change jdbc into session and don't need to create metadata (#1914)

上级 6d8cdd0f
......@@ -72,14 +72,7 @@ For the latest information about IoTDB, please visit [IoTDB official website](ht
- [Stop IoTDB](#stop-iotdb)
- [Only build server](#only-build-server)
- [Only build cli](#only-build-cli)
- [Usage of import-csv.sh](#usage-of-import-csvsh)
- [Create metadata](#create-metadata)
- [An example of import csv file](#an-example-of-import-csv-file)
- [Run import shell](#run-import-shell)
- [Error data file](#error-data-file)
- [Usage of export-csv.sh](#usage-of-export-csvsh)
- [Run export shell](#run-export-shell)
- [Input query](#input-query)
- [Usage of CSV Import and Export Tool](#usage-of-csv-import-and-export-tool)
<!-- /TOC -->
......@@ -333,59 +326,9 @@ Under the root path of iotdb:
After being built, the IoTDB cli is located at the folder "cli/target/iotdb-cli-{project.version}".
## Usage of import-csv.sh
### Create metadata
```
SET STORAGE GROUP TO root.fit.d1;
SET STORAGE GROUP TO root.fit.d2;
SET STORAGE GROUP TO root.fit.p;
CREATE TIMESERIES root.fit.d1.s1 WITH DATATYPE=INT32,ENCODING=RLE;
CREATE TIMESERIES root.fit.d1.s2 WITH DATATYPE=TEXT,ENCODING=PLAIN;
CREATE TIMESERIES root.fit.d2.s1 WITH DATATYPE=INT32,ENCODING=RLE;
CREATE TIMESERIES root.fit.d2.s3 WITH DATATYPE=INT32,ENCODING=RLE;
CREATE TIMESERIES root.fit.p.s1 WITH DATATYPE=INT32,ENCODING=RLE;
```
### An example of import csv file
```
Time,root.fit.d1.s1,root.fit.d1.s2,root.fit.d2.s1,root.fit.d2.s3,root.fit.p.s1
1,100,'hello',200,300,400
2,500,'world',600,700,800
3,900,'IoTDB',1000,1100,1200
```
### Run import shell
```
# Unix/OS X
> tools/import-csv.sh -h <ip> -p <port> -u <username> -pw <password> -f <xxx.csv>
# Windows
> tools\import-csv.bat -h <ip> -p <port> -u <username> -pw <password> -f <xxx.csv>
```
### Error data file
`csvInsertError.error`
## Usage of export-csv.sh
### Run export shell
```
# Unix/OS X
> tools/export-csv.sh -h <ip> -p <port> -u <username> -pw <password> -td <directory> [-tf <time-format>]
# Windows
> tools\export-csv.bat -h <ip> -p <port> -u <username> -pw <password> -td <directory> [-tf <time-format>]
```
### Input query
```
select * from root.fit.d1
```
# Usage of CSV Import and Export Tool
see [Usage of CSV Import and Export Tool](https://iotdb.apache.org/UserGuide/System%20Tools/CSV%20Tool.html)
# Frequent Questions for Compiling
see [Frequent Questions when Compiling the Source Code](https://iotdb.apache.org/Development/ContributeGuide.html#_Frequent-Questions-when-Compiling-the-Source-Code)
......
......@@ -70,14 +70,7 @@ IoTDB的主要特点如下:
- [停止 IoTDB](#停止-iotdb)
- [只编译 server](#只编译-server)
- [只编译 cli](#只编译-cli)
- [使用 import-csv.sh](#使用-import-csvsh)
- [创建元数据](#创建元数据)
- [从 csv 文件导入数据的示例](#从-csv-文件导入数据的示例)
- [运行 import shell](#运行-import-shell)
- [错误的数据文件](#错误的数据文件)
- [使用 export-csv.sh](#使用-export-csvsh)
- [运行 export shell](#运行-export-shell)
- [执行查询](#执行查询)
- [导入导出CSV工具](#导入导出CSV工具)
<!-- /TOC -->
......@@ -333,61 +326,9 @@ server 可以使用 "ctrl-C" 或者执行下面的脚本:
编译完成后, IoTDB cli 将生成在 "cli/target/iotdb-cli-{project.version}".
## 使用 import-csv.sh
### 创建元数据
```
SET STORAGE GROUP TO root.fit.d1;
SET STORAGE GROUP TO root.fit.d2;
SET STORAGE GROUP TO root.fit.p;
CREATE TIMESERIES root.fit.d1.s1 WITH DATATYPE=INT32,ENCODING=RLE;
CREATE TIMESERIES root.fit.d1.s2 WITH DATATYPE=TEXT,ENCODING=PLAIN;
CREATE TIMESERIES root.fit.d2.s1 WITH DATATYPE=INT32,ENCODING=RLE;
CREATE TIMESERIES root.fit.d2.s3 WITH DATATYPE=INT32,ENCODING=RLE;
CREATE TIMESERIES root.fit.p.s1 WITH DATATYPE=INT32,ENCODING=RLE;
```
### 从 csv 文件导入数据的示例
```
Time,root.fit.d1.s1,root.fit.d1.s2,root.fit.d2.s1,root.fit.d2.s3,root.fit.p.s1
1,100,'hello',200,300,400
2,500,'world',600,700,800
3,900,'IoTDB',1000,1100,1200
```
### 运行 import shell
```
# Unix/OS X
> tools/import-csv.sh -h <ip> -p <port> -u <username> -pw <password> -f <xxx.csv>
# Windows
> tools\import-csv.bat -h <ip> -p <port> -u <username> -pw <password> -f <xxx.csv>
```
### 错误的数据文件
`csvInsertError.error`
## 使用 export-csv.sh
### 运行 export shell
```
# Unix/OS X
> tools/export-csv.sh -h <ip> -p <port> -u <username> -pw <password> -td <directory> [-tf <time-format>]
# Windows
> tools\export-csv.bat -h <ip> -p <port> -u <username> -pw <password> -td <directory> [-tf <time-format>]
```
### 执行查询
```
select * from root.fit.d1
```
# 导入导出CSV工具
查看 [导入导出CSV工具](https://iotdb.apache.org/zh/UserGuide/System%20Tools/CSV%20Tool.html)
# 常见编译错误
查看 [常见编译错误](https://iotdb.apache.org/zh/Development/ContributeGuide.html#%E5%B8%B8%E8%A7%81%E7%BC%96%E8%AF%91%E9%94%99%E8%AF%AF)
......@@ -401,4 +342,4 @@ select * from root.fit.d1
* 添加好友 tietouqiao,我们会邀请您进群
获取更多内容,请查看 [加入社区](https://github.com/apache/iotdb/issues/1995)
\ No newline at end of file
获取更多内容,请查看 [加入社区](https://github.com/apache/iotdb/issues/1995)
......@@ -37,7 +37,7 @@
<dependencies>
<dependency>
<groupId>org.apache.iotdb</groupId>
<artifactId>iotdb-jdbc</artifactId>
<artifactId>iotdb-session</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
......@@ -47,6 +47,11 @@
<type>test-jar</type>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.iotdb</groupId>
<artifactId>iotdb-jdbc</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>commons-cli</groupId>
<artifactId>commons-cli</artifactId>
......
......@@ -536,14 +536,9 @@ public abstract class AbstractCli {
+ "Noted that your file path cannot contain any space character)");
return;
}
try {
println(cmd.split(" ")[1]);
ImportCsv.importCsvFromFile(host, port, username, password, cmd.split(" ")[1],
connection.getTimeZone());
} catch (SQLException e) {
println(String.format("Failed to import from %s because %s",
cmd.split(" ")[1], e.getMessage()));
}
println(cmd.split(" ")[1]);
ImportCsv.importCsvFromFile(host, port, username, password, cmd.split(" ")[1],
connection.getTimeZone());
}
private static void executeQuery(IoTDBConnection connection, String cmd) {
......
......@@ -22,10 +22,12 @@ import java.io.IOException;
import java.time.ZoneId;
import jline.console.ConsoleReader;
import org.apache.commons.cli.CommandLine;
import org.apache.commons.cli.Option;
import org.apache.commons.cli.Options;
import org.apache.iotdb.exception.ArgsErrorException;
import org.apache.iotdb.jdbc.IoTDBConnection;
import org.apache.iotdb.jdbc.IoTDBSQLException;
import org.apache.thrift.TException;
import org.apache.iotdb.rpc.IoTDBConnectionException;
import org.apache.iotdb.rpc.StatementExecutionException;
import org.apache.iotdb.session.Session;
public abstract class AbstractCsvTool {
......@@ -69,7 +71,7 @@ public abstract class AbstractCsvTool {
protected static String timeZoneID;
protected static String timeFormat;
protected static IoTDBConnection connection;
protected static Session session;
AbstractCsvTool() {}
......@@ -85,11 +87,11 @@ public abstract class AbstractCsvTool {
return str;
}
protected static void setTimeZone() throws IoTDBSQLException, TException {
protected static void setTimeZone() throws IoTDBConnectionException, StatementExecutionException {
if (timeZoneID != null) {
connection.setTimeZone(timeZoneID);
session.setTimeZone(timeZoneID);
}
zoneId = ZoneId.of(connection.getTimeZone());
zoneId = ZoneId.of(session.getTimeZone());
}
protected static void parseBasicParams(CommandLine commandLine, ConsoleReader reader)
......@@ -110,8 +112,32 @@ public abstract class AbstractCsvTool {
return true;
}
}
System.out.println(String.format("Input time format %s is not supported, "
+ "please input like yyyy-MM-dd\\ HH:mm:ss.SSS or yyyy-MM-dd'T'HH:mm:ss.SSS", timeFormat));
System.out.printf("Input time format %s is not supported, "
+ "please input like yyyy-MM-dd\\ HH:mm:ss.SSS or yyyy-MM-dd'T'HH:mm:ss.SSS%n", timeFormat);
return false;
}
protected static Options createNewOptions() {
Options options = new Options();
Option opHost = Option.builder(HOST_ARGS).longOpt(HOST_NAME).required().argName(HOST_NAME)
.hasArg()
.desc("Host Name (required)").build();
options.addOption(opHost);
Option opPort = Option.builder(PORT_ARGS).longOpt(PORT_NAME).required().argName(PORT_NAME)
.hasArg()
.desc("Port (required)").build();
options.addOption(opPort);
Option opUsername = Option.builder(USERNAME_ARGS).longOpt(USERNAME_NAME).required()
.argName(USERNAME_NAME)
.hasArg().desc("Username (required)").build();
options.addOption(opUsername);
Option opPassword = Option.builder(PASSWORD_ARGS).longOpt(PASSWORD_NAME).optionalArg(true)
.argName(PASSWORD_NAME).hasArg().desc("Password (optional)").build();
options.addOption(opPassword);
return options;
}
}
......@@ -25,16 +25,9 @@ import java.io.File;
import java.io.FileReader;
import java.io.FileWriter;
import java.io.IOException;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.ResultSetMetaData;
import java.sql.SQLException;
import java.sql.Statement;
import java.sql.Types;
import java.time.Instant;
import java.time.ZonedDateTime;
import java.time.format.DateTimeFormatter;
import java.util.ArrayList;
import java.util.List;
import jline.console.ConsoleReader;
import org.apache.commons.cli.CommandLine;
......@@ -46,9 +39,13 @@ import org.apache.commons.cli.Options;
import org.apache.commons.cli.ParseException;
import org.apache.iotdb.cli.AbstractCli;
import org.apache.iotdb.exception.ArgsErrorException;
import org.apache.iotdb.jdbc.Config;
import org.apache.iotdb.jdbc.IoTDBConnection;
import org.apache.thrift.TException;
import org.apache.iotdb.rpc.IoTDBConnectionException;
import org.apache.iotdb.rpc.StatementExecutionException;
import org.apache.iotdb.session.Session;
import org.apache.iotdb.session.SessionDataSet;
import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
import org.apache.iotdb.tsfile.read.common.Field;
import org.apache.iotdb.tsfile.read.common.RowRecord;
/**
* Export CSV file.
......@@ -75,20 +72,16 @@ public class ExportCsv extends AbstractCsvTool {
private static final int EXPORT_PER_LINE_COUNT = 10000;
private static String TIMESTAMP_PRECISION = "ms";
private static List<Integer> typeList = new ArrayList<>();
/**
* main function of export csv tool.
*/
public static void main(String[] args) throws IOException, SQLException {
public static void main(String[] args) throws IOException {
Options options = createOptions();
HelpFormatter hf = new HelpFormatter();
hf.setOptionComparator(null); // avoid reordering
hf.setWidth(MAX_HELP_CONSOLE_WIDTH);
CommandLine commandLine;
CommandLineParser parser = new DefaultParser();
hf.setOptionComparator(null); // avoid reordering
hf.setWidth(MAX_HELP_CONSOLE_WIDTH);
if (args == null || args.length == 0) {
System.out.println("Too few params input, please check the following hint.");
......@@ -116,13 +109,11 @@ public class ExportCsv extends AbstractCsvTool {
if (!checkTimeFormat()) {
return;
}
Class.forName(Config.JDBC_DRIVER_NAME);
String sqlFile = commandLine.getOptionValue(SQL_FILE_ARGS);
String sql;
connection = (IoTDBConnection) DriverManager
.getConnection(Config.IOTDB_URL_PREFIX + host + ":" + port + "/", username, password);
session = new Session(host, Integer.parseInt(port), username, password);
session.open(false);
setTimeZone();
if (sqlFile == null) {
......@@ -134,21 +125,21 @@ public class ExportCsv extends AbstractCsvTool {
} else {
dumpFromSqlFile(sqlFile);
}
} catch (ClassNotFoundException e) {
System.out.println("Failed to export data because cannot find IoTDB JDBC Driver, "
+ "please check whether you have imported driver or not: " + e.getMessage());
} catch (TException e) {
System.out.println("Encounter an error when connecting to server, because " + e.getMessage());
} catch (SQLException e) {
System.out.println("Encounter an error when exporting data, error is: " + e.getMessage());
} catch (IOException e) {
System.out.println("Failed to operate on file, because " + e.getMessage());
} catch (ArgsErrorException e) {
System.out.println("Invalid args: " + e.getMessage());
} catch (IoTDBConnectionException | StatementExecutionException e) {
System.out.println("Connect failed because " + e.getMessage());
} finally {
reader.close();
if (connection != null) {
connection.close();
if (session != null) {
try {
session.close();
} catch (IoTDBConnectionException e) {
System.out
.println("Encounter an error when closing session, error is: " + e.getMessage());
}
}
}
}
......@@ -176,26 +167,7 @@ public class ExportCsv extends AbstractCsvTool {
* @return object Options
*/
private static Options createOptions() {
Options options = new Options();
Option opHost = Option.builder(HOST_ARGS).longOpt(HOST_NAME).required().argName(HOST_NAME)
.hasArg()
.desc("Host Name (required)").build();
options.addOption(opHost);
Option opPort = Option.builder(PORT_ARGS).longOpt(PORT_NAME).required().argName(PORT_NAME)
.hasArg()
.desc("Port (required)").build();
options.addOption(opPort);
Option opUsername = Option.builder(USERNAME_ARGS).longOpt(USERNAME_NAME).required()
.argName(USERNAME_NAME)
.hasArg().desc("Username (required)").build();
options.addOption(opUsername);
Option opPassword = Option.builder(PASSWORD_ARGS).longOpt(PASSWORD_NAME).optionalArg(true)
.argName(PASSWORD_NAME).hasArg().desc("Password (optional)").build();
options.addOption(opPassword);
Options options = createNewOptions();
Option opTargetFile = Option.builder(TARGET_DIR_ARGS).required().argName(TARGET_DIR_NAME)
.hasArg()
......@@ -234,12 +206,7 @@ public class ExportCsv extends AbstractCsvTool {
String sql;
int index = 0;
while ((sql = reader.readLine()) != null) {
try {
dumpResult(sql, index);
} catch (SQLException e) {
System.out
.println("Cannot dump data for statement " + sql + ", because : " + e.getMessage());
}
dumpResult(sql, index);
index++;
}
}
......@@ -248,12 +215,10 @@ public class ExportCsv extends AbstractCsvTool {
/**
* Dump files from database to CSV file.
*
* @param sql export the result of executing the sql
* @param sql export the result of executing the sql
* @param index use to create dump file name
* @throws SQLException if SQL is not valid
*/
private static void dumpResult(String sql, int index)
throws SQLException {
private static void dumpResult(String sql, int index) {
final String path = targetDirectory + targetFile + index + ".csv";
File tf = new File(path);
......@@ -263,82 +228,73 @@ public class ExportCsv extends AbstractCsvTool {
return;
}
} catch (IOException e) {
System.out.println("Cannot create dump file " + path + "because: " + e.getMessage());
System.out.println("Cannot create dump file " + path + " " + "because: " + e.getMessage());
return;
}
System.out.println("Start to export data from sql statement: " + sql);
try (Statement statement = connection.createStatement();
ResultSet rs = statement.executeQuery(sql);
BufferedWriter bw = new BufferedWriter(new FileWriter(tf))) {
ResultSetMetaData metadata = rs.getMetaData();
try (BufferedWriter bw = new BufferedWriter(new FileWriter(tf))) {
SessionDataSet sessionDataSet = session.executeQueryStatement(sql);
long startTime = System.currentTimeMillis();
int count = metadata.getColumnCount();
// write data in csv file
writeMetadata(bw, count, metadata);
writeMetadata(bw, sessionDataSet.getColumnNames());
int line = writeResultSet(rs, bw, count);
int line = writeResultSet(sessionDataSet, bw);
System.out
.println(String.format("Statement [%s] has dumped to file %s successfully! It costs "
+ "%dms to export %d lines.", sql, path, System.currentTimeMillis() - startTime,
line));
} catch (IOException e) {
.printf("Statement [%s] has dumped to file %s successfully! It costs "
+ "%dms to export %d lines.%n", sql, path, System.currentTimeMillis() - startTime,
line);
} catch (IOException | StatementExecutionException | IoTDBConnectionException e) {
System.out.println("Cannot dump result because: " + e.getMessage());
}
}
private static void writeMetadata(BufferedWriter bw, int count, ResultSetMetaData metadata)
throws SQLException, IOException {
for (int i = 1; i <= count; i++) {
if (i < count) {
bw.write(metadata.getColumnLabel(i) + ",");
} else {
bw.write(metadata.getColumnLabel(i) + "\n");
}
typeList.add(metadata.getColumnType(i));
private static void writeMetadata(BufferedWriter bw, List<String> columnNames)
throws IOException {
if (!columnNames.get(0).equals("Time")) {
bw.write("Time" + ",");
}
for (int i = 0; i < columnNames.size() - 1; i++) {
bw.write(columnNames.get(i) + ",");
}
bw.write(columnNames.get(columnNames.size() - 1) + "\n");
}
private static int writeResultSet(ResultSet rs, BufferedWriter bw, int count)
throws SQLException, IOException {
private static int writeResultSet(SessionDataSet rs, BufferedWriter bw)
throws IOException, StatementExecutionException, IoTDBConnectionException {
int line = 0;
long timestamp = System.currentTimeMillis();
while (rs.next()) {
if (rs.getString(1) == null ||
"null".equalsIgnoreCase(rs.getString(1))) {
bw.write(",");
} else {
writeTime(rs, bw);
writeValue(rs, count, bw);
}
while (rs.hasNext()) {
RowRecord rowRecord = rs.next();
List<Field> fields = rowRecord.getFields();
writeTime(rowRecord.getTimestamp(), bw);
writeValue(fields, bw);
line++;
if (line % EXPORT_PER_LINE_COUNT == 0) {
long tmp = System.currentTimeMillis();
System.out.println(
String.format("%d lines have been exported, it takes %dms", line, (tmp - timestamp)));
System.out.printf("%d lines have been exported, it takes %dms%n", line, (tmp - timestamp));
timestamp = tmp;
}
}
return line;
}
private static void writeTime(ResultSet rs, BufferedWriter bw) throws SQLException, IOException {
private static void writeTime(Long time, BufferedWriter bw) throws IOException {
ZonedDateTime dateTime;
String timestampPrecision = "ms";
switch (timeFormat) {
case "default":
long timestamp = rs.getLong(1);
String str = AbstractCli
.parseLongToDateWithPrecision(DateTimeFormatter.ISO_OFFSET_DATE_TIME, timestamp, zoneId,
TIMESTAMP_PRECISION);
.parseLongToDateWithPrecision(DateTimeFormatter.ISO_OFFSET_DATE_TIME, time, zoneId,
timestampPrecision);
bw.write(str + ",");
break;
case "timestamp":
case "long":
case "nubmer":
bw.write(rs.getLong(1) + ",");
bw.write(time + ",");
break;
default:
dateTime = ZonedDateTime.ofInstant(Instant.ofEpochMilli(rs.getLong(1)),
dateTime = ZonedDateTime.ofInstant(Instant.ofEpochMilli(time),
zoneId);
bw.write(dateTime.format(DateTimeFormatter.ofPattern(timeFormat)) + ",");
break;
......@@ -346,29 +302,49 @@ public class ExportCsv extends AbstractCsvTool {
}
@SuppressWarnings("squid:S3776") // Suppress high Cognitive Complexity warning
private static void writeValue(ResultSet rs, int count, BufferedWriter bw)
throws SQLException, IOException {
for (int j = 2; j <= count; j++) {
if (j < count) {
if ("null".equals(rs.getString(j))) {
bw.write(",");
} else {
if(typeList.get(j-1) == Types.VARCHAR) {
bw.write("\'" + rs.getString(j) + "\'"+ ",");
private static void writeValue(List<Field> fields, BufferedWriter bw) throws IOException {
for (int j = 0; j < fields.size() - 1; j++) {
String value = fields.get(j).getStringValue();
if ("null".equalsIgnoreCase(value)) {
bw.write(",");
} else {
if (fields.get(j).getDataType() == TSDataType.TEXT) {
int location = value.indexOf("\"");
if (location > -1) {
if (location == 0 || value.charAt(location - 1) != '\\') {
bw.write("\"" + value.replace("\"", "\\\"") + "\",");
} else {
bw.write("\"" + value + "\",");
}
} else if (value.contains(",")) {
bw.write("\"" + value + "\",");
} else {
bw.write(rs.getString(j) + ",");
bw.write(value + ",");
}
}
} else {
if ("null".equals(rs.getString(j))) {
bw.write("\n");
} else {
if(typeList.get(j-1) == Types.VARCHAR) {
bw.write("\'" + rs.getString(j) + "\'"+ "\n");
bw.write(value + ",");
}
}
}
String lastValue = fields.get(fields.size() - 1).getStringValue();
if ("null".equalsIgnoreCase(lastValue)) {
bw.write("\n");
} else {
if (fields.get(fields.size() - 1).getDataType() == TSDataType.TEXT) {
int location = lastValue.indexOf("\"");
if (location > -1) {
if (location == 0 || lastValue.charAt(location - 1) != '\\') {
bw.write("\"" + lastValue.replace("\"", "\\\"") + "\"\n");
} else {
bw.write(rs.getString(j) + "\n");
bw.write("\"" + lastValue + "\"\n");
}
} else if (lastValue.contains(",")) {
bw.write("\"" + lastValue + "\"\n");
} else {
bw.write(lastValue + "\n");
}
} else {
bw.write(lastValue + "\n");
}
}
}
......
......@@ -38,6 +38,10 @@ public abstract class AbstractScript {
if (line == null) {
break;
} else {
// remove thing after "connection refused", only for test
if(line.contains("Connection refused")) {
line = line.substring(0, line.indexOf("Connection refused") + "Connection refused".length());
}
outputList.add(line);
}
}
......
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
package org.apache.iotdb.tool;
import org.junit.Assert;
import org.junit.Test;
public class CsvLineSplitTest {
@Test
public void testSplit() {
Assert.assertArrayEquals(new String[]{"", "a", "b", "c", "\\\""}, ImportCsv.splitCsvLine(",a,b,c,\"\\\"\""));
Assert.assertArrayEquals(new String[]{"", "a", "b", "\\'"}, ImportCsv.splitCsvLine(",a,b,\"\\'\""));
}
}
......@@ -28,11 +28,11 @@ import org.junit.Test;
public class ExportCsvTestIT extends AbstractScript{
@Before
public void setUp() throws Exception {
public void setUp() {
}
@After
public void tearDown() throws Exception {
public void tearDown() {
}
@Test
......@@ -50,8 +50,8 @@ public class ExportCsvTestIT extends AbstractScript{
final String[] output = {"````````````````````````````````````````````````",
"Starting IoTDB Client Export Script",
"````````````````````````````````````````````````",
"Encounter an error when exporting data, error is: Connection Error, "
+ "please check whether the network is available or the server has started."};
"Connect failed because org.apache.thrift.transport.TTransportException: "
+ "java.net.ConnectException: Connection refused"};
String dir = getCliPath();
ProcessBuilder builder = new ProcessBuilder("cmd.exe", "/c",
dir + File.separator + "tools" + File.separator + "export-csv.bat",
......@@ -64,8 +64,8 @@ public class ExportCsvTestIT extends AbstractScript{
final String[] output = {"------------------------------------------",
"Starting IoTDB Client Export Script",
"------------------------------------------",
"Encounter an error when exporting data, error is: Connection Error, "
+ "please check whether the network is available or the server has started."};
"Connect failed because org.apache.thrift.transport.TTransportException: "
+ "java.net.ConnectException: Connection refused"};
String dir = getCliPath();
ProcessBuilder builder = new ProcessBuilder("sh",
dir + File.separator + "tools" + File.separator + "export-csv.sh",
......
......@@ -28,11 +28,11 @@ import org.junit.Test;
public class ImportCsvTestIT extends AbstractScript {
@Before
public void setUp() throws Exception {
public void setUp() {
}
@After
public void tearDown() throws Exception {
public void tearDown() {
}
@Test
......@@ -50,8 +50,8 @@ public class ImportCsvTestIT extends AbstractScript {
final String[] output = {"````````````````````````````````````````````````",
"Starting IoTDB Client Import Script",
"````````````````````````````````````````````````",
"Encounter an error when importing data, error is: Connection Error, please check whether "
+ "the network is available or the server has started."};
"Encounter an error when connecting to server, because org.apache.thrift.transport.TTransportException: "
+ "java.net.ConnectException: Connection refused"};
String dir = getCliPath();
ProcessBuilder builder = new ProcessBuilder("cmd.exe", "/c",
dir + File.separator + "tools" + File.separator + "import-csv.bat",
......@@ -64,8 +64,8 @@ public class ImportCsvTestIT extends AbstractScript {
final String[] output = {"------------------------------------------",
"Starting IoTDB Client Import Script",
"------------------------------------------",
"Encounter an error when importing data, error is: Connection Error, please check whether "
+ "the network is available or the server has started."};
"Encounter an error when connecting to server, because org.apache.thrift.transport.TTransportException: "
+ "java.net.ConnectException: Connection refused"};
String dir = getCliPath();
ProcessBuilder builder = new ProcessBuilder("sh",
dir + File.separator + "tools" + File.separator + "import-csv.sh",
......
......@@ -49,7 +49,7 @@ public abstract class AbstractScript {
p.destroy();
System.out.println("Expected output:");
System.out.println("should contains:");
for (String s : output) {
System.out.println(s);
}
......@@ -59,7 +59,7 @@ public abstract class AbstractScript {
System.out.println(out);
}
assertTrue(actualOutput.get(actualOutput.size() - 1).startsWith(output[output.length - 1]));
assertTrue(actualOutput.get(actualOutput.size() - 1).contains(output[output.length - 1]));
}
protected String getCliPath() {
......
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
package org.apache.iotdb.cross.tests.tools.importCsv;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertTrue;
import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileReader;
import java.io.FileWriter;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import org.apache.iotdb.db.utils.EnvironmentUtils;
import org.apache.iotdb.rpc.IoTDBConnectionException;
import org.apache.iotdb.rpc.StatementExecutionException;
import org.apache.iotdb.session.Session;
import org.junit.After;
import org.junit.Before;
import org.junit.Test;
public class ExportCsvTestIT extends AbstractScript{
private final String SQL_FILE = "target" + File.separator + "sql.txt";
private final String EXPORT_FILE = "target" + File.separator + "dump0.csv";
@Before
public void setUp() {
EnvironmentUtils.closeStatMonitor();
EnvironmentUtils.envSetUp();
}
@After
public void tearDown() throws Exception {
EnvironmentUtils.cleanEnv();
}
@Override
protected void testOnWindows() throws IOException {
final String[] output = {
"------------------------------------------",
"Starting IoTDB Client Export Script",
"------------------------------------------",
"Start to export data from sql statement",
"successfully",
};
String dir = getCliPath();
ProcessBuilder builder = new ProcessBuilder("cmd.exe", "/c",
dir + File.separator + "tools" + File.separator + "export-csv.bat",
"-h", "127.0.0.1", "-p", "6667", "-u", "root", "-pw", "root", "-td", "./target",
"-s", SQL_FILE);
testOutput(builder, output);
}
@Override
protected void testOnUnix() throws IOException {
final String[] output = {
"------------------------------------------",
"Starting IoTDB Client Export Script",
"------------------------------------------",
"Start to export data from sql statement",
"successfully",
};
String dir = getCliPath();
ProcessBuilder builder = new ProcessBuilder("sh",
dir + File.separator + "tools" + File.separator + "export-csv.sh",
"-h", "127.0.0.1", "-p", "6667", "-u", "root", "-pw", "root", "-td", "./target",
"-s", SQL_FILE);
testOutput(builder, output);
}
private boolean generateSQLFile(String[] sql) {
BufferedWriter writer;
try {
writer = new BufferedWriter(new FileWriter(SQL_FILE));
writer.write("");
for (String s : sql) {
writer.write(s);
writer.newLine();
}
writer.flush();
writer.close();
return true;
} catch (IOException e) {
System.out.println("failed to create test csv");
}
return false;
}
@Test
public void testRawDataQuery() throws IOException, StatementExecutionException, IoTDBConnectionException {
final String[] expectCsv = new String[]{"Time,root.sg1.d1.s3,root.sg1.d1.s1,root.sg1.d1.s2",
"abbe's,1.0,\"\\\"abc\\\",aa\""};
prepareData();
String os = System.getProperty("os.name").toLowerCase();
String[] sql = {"select * from root"};
assertTrue(generateSQLFile(sql));
if (os.startsWith("windows")) {
testOnWindows();
} else {
testOnUnix();
}
FileReader fileReader = new FileReader(EXPORT_FILE);
BufferedReader br = new BufferedReader(fileReader);
String line = br.readLine();
int i = 0;
while(line != null) {
if(i == 0) {
assertEquals(expectCsv[i], line);
} else {
String lineWithoutTime = line.substring(line.indexOf(',') + 1);
assertEquals(expectCsv[i], lineWithoutTime);
}
i++;
line = br.readLine();
}
File file = new File(EXPORT_FILE);
if (file.exists()) {
file.delete();
}
}
@Test
public void testAggregationQuery() throws StatementExecutionException, IoTDBConnectionException, IOException {
final String[] expectCsv = new String[]{"Time,count(root.sg1.d1.s3),count(root.sg1.d1.s1),count(root.sg1.d1.s2)",
"1,1,1"};
prepareData();
String os = System.getProperty("os.name").toLowerCase();
String[] sql = {"select count(*) from root"};
generateSQLFile(sql);
if (os.startsWith("windows")) {
testOnWindows();
} else {
testOnUnix();
}
FileReader fileReader = new FileReader(EXPORT_FILE);
BufferedReader br = new BufferedReader(fileReader);
String line = br.readLine();
int i = 0;
while(line != null) {
if(i == 0) {
assertEquals(expectCsv[i], line);
} else {
String lineWithoutTime = line.substring(line.indexOf(',') + 1);
assertEquals(expectCsv[i], lineWithoutTime);
}
i++;
line = br.readLine();
}
File file = new File(EXPORT_FILE);
if (file.exists()) {
file.delete();
}
}
private void prepareData() throws IoTDBConnectionException, StatementExecutionException {
Session session = new Session("127.0.0.1", 6667, "root", "root");
session.open();
String deviceId = "root.sg1.d1";
List<String> measurements = new ArrayList<>();
measurements.add("s1");
measurements.add("s2");
measurements.add("s3");
List<String> values = new ArrayList<>();
values.add("1.0");
values.add("\"abc\",aa");
values.add("abbe's");
session.insertRecord(deviceId, 1L, measurements, values);
}
}
......@@ -20,11 +20,8 @@
package org.apache.iotdb.cross.tests.tools.importCsv;
import org.apache.iotdb.db.utils.EnvironmentUtils;
import org.apache.iotdb.jdbc.Config;
import org.junit.After;
import org.junit.Before;
import org.junit.Test;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertTrue;
import java.io.BufferedWriter;
import java.io.File;
......@@ -32,16 +29,23 @@ import java.io.FileWriter;
import java.io.IOException;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.ResultSetMetaData;
import java.sql.SQLException;
import java.sql.Statement;
import static org.junit.Assert.assertTrue;
import java.util.ArrayList;
import java.util.List;
import org.apache.iotdb.db.utils.EnvironmentUtils;
import org.apache.iotdb.jdbc.Config;
import org.junit.After;
import org.junit.Before;
import org.junit.Test;
public class ImportCsvTestIT extends AbstractScript {
private final String CSV_FILE = "target" + File.separator + "test.csv";
private static String[] sqls = new String[]{
private static final String[] sqls = new String[]{
"SET STORAGE GROUP TO root.fit.d1",
"SET STORAGE GROUP TO root.fit.d2",
"SET STORAGE GROUP TO root.fit.p",
......@@ -53,11 +57,9 @@ public class ImportCsvTestIT extends AbstractScript {
};
@Before
public void setUp() throws Exception {
public void setUp() {
EnvironmentUtils.closeStatMonitor();
EnvironmentUtils.envSetUp();
createSchema();
}
@After
......@@ -80,7 +82,23 @@ public class ImportCsvTestIT extends AbstractScript {
}
@Test
public void test() throws IOException {
public void test() throws IOException, ClassNotFoundException {
createSchema();
String os = System.getProperty("os.name").toLowerCase();
assertTrue(generateTestCSV());
if (os.startsWith("windows")) {
testOnWindows();
} else {
testOnUnix();
}
File file = new File(CSV_FILE);
if (file.exists()) {
file.delete();
}
}
@Test
public void testWithoutCreateSchema() throws IOException, ClassNotFoundException {
String os = System.getProperty("os.name").toLowerCase();
assertTrue(generateTestCSV());
if (os.startsWith("windows")) {
......@@ -89,6 +107,71 @@ public class ImportCsvTestIT extends AbstractScript {
testOnUnix();
}
File file = new File(CSV_FILE);
Class.forName(Config.JDBC_DRIVER_NAME);
try (Connection connection = DriverManager
.getConnection(Config.IOTDB_URL_PREFIX + "127.0.0.1:6667/", "root", "root");
Statement statement = connection.createStatement()) {
if(statement.execute("select * from root")) {
ResultSet resultSet = statement.getResultSet();
testResult(resultSet, 6,3);
}
} catch (Exception e) {
e.printStackTrace();
}
if (file.exists()) {
file.delete();
}
}
@Test
public void testBigCsvFile() throws IOException, ClassNotFoundException {
String os = System.getProperty("os.name").toLowerCase();
assertTrue(generateBigCsvFile());
if (os.startsWith("windows")) {
testOnWindows();
} else {
testOnUnix();
}
File file = new File(CSV_FILE);
Class.forName(Config.JDBC_DRIVER_NAME);
try (Connection connection = DriverManager
.getConnection(Config.IOTDB_URL_PREFIX + "127.0.0.1:6667/", "root", "root");
Statement statement = connection.createStatement()) {
if(statement.execute("select s1 from root.fit.d1")) {
ResultSet resultSet = statement.getResultSet();
testResult(resultSet, 2,25000);
}
} catch (Exception e) {
e.printStackTrace();
}
if (file.exists()) {
file.delete();
}
}
private static void testResult(ResultSet resultSet, int expectedColumnNumber, int expectedRowNumber) throws SQLException {
if (resultSet != null) {
final ResultSetMetaData metaData = resultSet.getMetaData();
final int columnCount = metaData.getColumnCount();
assertEquals(expectedColumnNumber, columnCount);
int actualRowNumber = 0;
while (resultSet.next()) {
actualRowNumber++;
}
assertEquals(expectedRowNumber, actualRowNumber);
}
}
@Test
public void testImportHeaderCSV() throws IOException {
String os = System.getProperty("os.name").toLowerCase();
assertTrue(generateHeaderTestCSV());
if (os.startsWith("windows")) {
testOnWindows();
} else {
testOnUnix();
}
File file = new File(CSV_FILE);
if (file.exists()) {
file.delete();
}
......@@ -98,8 +181,51 @@ public class ImportCsvTestIT extends AbstractScript {
String[] csvText = {
"Time,root.fit.d1.s1,root.fit.d1.s2,root.fit.d2.s1,root.fit.d2.s3,root.fit.p.s1",
"1,100,'hello',200,300,400",
"2,500,'world',600,700,800",
"3,900,'IoTDB',1000,1100,1200"};
"2,500,'',600,700,800",
"3,900,'Io\"TDB',1000,1100,1200"};
BufferedWriter writer;
try {
writer = new BufferedWriter(new FileWriter(CSV_FILE));
writer.write("");
for (String s : csvText) {
writer.write(s);
writer.newLine();
}
writer.flush();
writer.close();
return true;
} catch (IOException e) {
System.out.println("failed to create test csv");
}
return false;
}
private boolean generateBigCsvFile() {
List<String> csvText = new ArrayList<>();
csvText.add("Time,root.fit.d1.s1,root.fit.d1.s2,root.fit.d2.s1");
for(int i = 0; i < 25000; i++) {
csvText.add(i+","+i+","+i+","+i);
}
BufferedWriter writer;
try {
writer = new BufferedWriter(new FileWriter(CSV_FILE));
writer.write("");
for (String s : csvText) {
writer.write(s);
writer.newLine();
}
writer.flush();
writer.close();
return true;
} catch (IOException e) {
System.out.println("failed to create test csv");
}
return false;
}
private boolean generateHeaderTestCSV() {
String[] csvText = {
"Time,root.fit.d1.\"s1\",root.fit.d1.s2,root.fit.d2.s1,root.fit.d2.s3,root.fit.p.s1"};
BufferedWriter writer;
try {
writer = new BufferedWriter(new FileWriter(CSV_FILE));
......
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
# Csv tool
Csv tool is that you can import csv file into IoTDB or export csv file from IoTDB.
## Usage of import-csv.sh
### Create metadata (optional)
```
SET STORAGE GROUP TO root.fit.d1;
SET STORAGE GROUP TO root.fit.d2;
SET STORAGE GROUP TO root.fit.p;
CREATE TIMESERIES root.fit.d1.s1 WITH DATATYPE=INT32,ENCODING=RLE;
CREATE TIMESERIES root.fit.d1.s2 WITH DATATYPE=TEXT,ENCODING=PLAIN;
CREATE TIMESERIES root.fit.d2.s1 WITH DATATYPE=INT32,ENCODING=RLE;
CREATE TIMESERIES root.fit.d2.s3 WITH DATATYPE=INT32,ENCODING=RLE;
CREATE TIMESERIES root.fit.p.s1 WITH DATATYPE=INT32,ENCODING=RLE;
```
IoTDB has the ability of type inference, so it is not necessary to create metadata before data import. However, we still recommend creating metadata before importing data using the CSV import tool, as this can avoid unnecessary type conversion errors.
### An example of import csv file
```
Time,root.fit.d1.s1,root.fit.d1.s2,root.fit.d2.s1,root.fit.d2.s3,root.fit.p.s1
1,100,hello,200,300,400
2,500,world,600,700,800
3,900,"hello, \"world\"",1000,1100,1200
```
> Note that the following special characters in fields need to be checked before importing:
> 1. `,` : fields containing `,` should be quoted by a pair of `"` or a pair of `'`.
> 2. `"` : `"` in fields should be replaced by `\"` or fields should be enclosed by `'`.
> 3. `'` : `'` in fields should be replaced by `\'` or fields should be enclosed by `"`.
### Run import shell
```
# Unix/OS X
> tools/import-csv.sh -h <ip> -p <port> -u <username> -pw <password> -f <xxx.csv>
# Windows
> tools\import-csv.bat -h <ip> -p <port> -u <username> -pw <password> -f <xxx.csv>
```
## Usage of export-csv.sh
### Run export shell
```
# Unix/OS X
> tools/export-csv.sh -h <ip> -p <port> -u <username> -pw <password> -td <directory> [-tf <time-format> -s <sqlfile>]
# Windows
> tools\export-csv.bat -h <ip> -p <port> -u <username> -pw <password> -td <directory> [-tf <time-format> -s <sqlfile>]
```
After running export shell, you need to input a data query, like `select * from root`. or specify a sql file. If you have multi sql in a sql file, sql should be split by new line character.
an example sql file:
```
select * from root.fit.d1
select * from root.sg1.d1
```
> Note that if fields exported by the export tool have the following special characters:
> 1. `,`: the field will be enclosed by `"`.
> 2. `"`: the field will be enclosed by `"` and the original characters `"` in the field will be replaced by `\"`.
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
# CSV 工具
Csv工具是您可以导入csv文件到IoTDB或从IoTDB导出csv文件。
## 使用 import-csv.sh
### 创建元数据(可选)
```
SET STORAGE GROUP TO root.fit.d1;
SET STORAGE GROUP TO root.fit.d2;
SET STORAGE GROUP TO root.fit.p;
CREATE TIMESERIES root.fit.d1.s1 WITH DATATYPE=INT32,ENCODING=RLE;
CREATE TIMESERIES root.fit.d1.s2 WITH DATATYPE=TEXT,ENCODING=PLAIN;
CREATE TIMESERIES root.fit.d2.s1 WITH DATATYPE=INT32,ENCODING=RLE;
CREATE TIMESERIES root.fit.d2.s3 WITH DATATYPE=INT32,ENCODING=RLE;
CREATE TIMESERIES root.fit.p.s1 WITH DATATYPE=INT32,ENCODING=RLE;
```
IoTDB具有类型推断的能力,因此在数据导入前创建元数据不是必须的。但我们仍然推荐在使用CSV导入工具导入数据前创建元数据,因为这可以避免不必要的类型转换错误。
### 从 csv 文件导入数据的示例
```
Time,root.fit.d1.s1,root.fit.d1.s2,root.fit.d2.s1,root.fit.d2.s3,root.fit.p.s1
1,100,hello,200,300,400
2,500,world,600,700,800
3,900,"hello, \"world\"",1000,1100,1200
```
> 注意,在导入数据前,需要特殊处理下列的字符:
> 1. `,` : 包含`,`的字段需要使用单引号或者双引号括起来
> 2. `"` : "字段中的`"`需要被替换成转义字符`\"`或者用`\'`将字段括起来。
> 3. `'` : "字段中的`'`需要被替换成转义字符`\'`或者用`\"`将字段括起来。
### 运行 import shell
```
# Unix/OS X
> tools/import-csv.sh -h <ip> -p <port> -u <username> -pw <password> -f <xxx.csv>
# Windows
> tools\import-csv.bat -h <ip> -p <port> -u <username> -pw <password> -f <xxx.csv>
```
## 使用 export-csv.sh
### 运行 export shell
```
# Unix/OS X
> tools/export-csv.sh -h <ip> -p <port> -u <username> -pw <password> -td <directory> [-tf <time-format> -s <sqlfile>]
# Windows
> tools\export-csv.bat -h <ip> -p <port> -u <username> -pw <password> -td <directory> [-tf <time-format> -s <sqlfile>]
```
在运行导出脚本之后,您需要输入一些查询或指定一些sql文件。如果在一个sql文件中有多个sql, sql应该被换行符分割。
一个sql文件例子
```
select * from root.fit.d1
select * from root.sg1.d1
```
> 注意,如果导出字段存在如下特殊字符:
> 1. `,` : 整个字段会被用`"`括起来。
> 2. `"` : 整个字段会被用`"`括起来且`"`会被替换为`\"`。
\ No newline at end of file
Markdown is supported
0% .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑!
想要评论请 注册