Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
apache
Iotdb
提交
742e6866
I
Iotdb
项目概览
apache
/
Iotdb
大约 1 年 前同步成功
通知
25
Star
3344
Fork
916
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
DevOps
流水线
流水线任务
计划
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
I
Iotdb
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
DevOps
DevOps
流水线
流水线任务
计划
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
流水线任务
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
未验证
提交
742e6866
编写于
3月 13, 2019
作者:
X
Xiangdong Huang
提交者:
GitHub
3月 13, 2019
浏览文件
操作
浏览文件
下载
差异文件
Merge pull request #93 from apache/jira39
[IOTDB-39]add autoRepair option for NativeRestorableIOWriter
上级
8f38c8bd
77206b27
变更
3
隐藏空白更改
内联
并排
Showing
3 changed file
with
61 addition
and
28 deletion
+61
-28
tsfile/src/main/java/org/apache/iotdb/tsfile/write/writer/NativeRestorableIOWriter.java
...e/iotdb/tsfile/write/writer/NativeRestorableIOWriter.java
+35
-11
tsfile/src/main/java/org/apache/iotdb/tsfile/write/writer/TsFileIOWriter.java
.../org/apache/iotdb/tsfile/write/writer/TsFileIOWriter.java
+1
-0
tsfile/src/test/java/org/apache/iotdb/tsfile/write/writer/NativeRestorableIOWriterTest.java
...tdb/tsfile/write/writer/NativeRestorableIOWriterTest.java
+25
-17
未找到文件。
tsfile/src/main/java/org/apache/iotdb/tsfile/write/writer/NativeRestorableIOWriter.java
浏览文件 @
742e6866
...
...
@@ -53,6 +53,15 @@ public class NativeRestorableIOWriter extends TsFileIOWriter {
}
public
NativeRestorableIOWriter
(
File
file
)
throws
IOException
{
this
(
file
,
true
);
}
/**
* @param file a given tsfile path you want to (continue to) write
* @param autoRepair whether auto repair the tsfile (if it is broken)
* @throws IOException if write failed, or the file is broken but autoRepair==false.
*/
public
NativeRestorableIOWriter
(
File
file
,
boolean
autoRepair
)
throws
IOException
{
super
();
long
fileSize
;
if
(!
file
.
exists
())
{
...
...
@@ -85,20 +94,28 @@ public class NativeRestorableIOWriter extends TsFileIOWriter {
boolean
newGroup
=
true
;
TsFileSequenceReader
reader
=
new
TsFileSequenceReader
(
file
.
getAbsolutePath
(),
false
);
if
(
fileSize
<=
magicStringBytes
.
length
)
{
LOGGER
.
debug
(
"{} only has magic header, does not worth to recover."
,
file
.
getAbsolutePath
());
reader
.
close
();
this
.
out
.
truncate
(
0
);
startFile
();
truncatedPosition
=
magicStringBytes
.
length
;
return
;
if
(
fileSize
<
magicStringBytes
.
length
)
{
reader
.
close
();
out
.
close
();
throw
new
IOException
(
String
.
format
(
"%s is not using TsFile format, and will be ignored..."
,
file
.
getAbsolutePath
()));
}
String
magic
=
reader
.
readHeadMagic
(
true
);
if
(!
magic
.
equals
(
new
String
(
magicStringBytes
)))
{
reader
.
close
();
out
.
close
();
throw
new
IOException
(
String
.
format
(
"%s is not using TsFile format, and will be ignored..."
,
file
.
getAbsolutePath
()));
}
if
(
reader
.
readTailMagic
().
equals
(
magic
))
{
if
(
fileSize
==
magicStringBytes
.
length
)
{
if
(!
autoRepair
)
{
reader
.
close
();
out
.
close
();
throw
new
IOException
(
String
.
format
(
"%s only has header, but does not allowed to be repaired..."
,
file
.
getAbsolutePath
()));
}
}
else
if
(
reader
.
readTailMagic
().
equals
(
magic
))
{
LOGGER
.
debug
(
"{} is an complete TsFile."
,
file
.
getAbsolutePath
());
canWrite
=
false
;
reader
.
close
();
...
...
@@ -192,10 +209,17 @@ public class NativeRestorableIOWriter extends TsFileIOWriter {
}
finally
{
//something wrong or all data is complete. We will discard current FileMetadata
// so that we can continue to write data into this tsfile.
LOGGER
.
info
(
"File {} has {} bytes, and will be truncated from {}."
,
file
.
getAbsolutePath
(),
file
.
length
(),
truncatedPosition
);
out
.
truncate
(
truncatedPosition
);
reader
.
close
();
if
(
autoRepair
)
{
LOGGER
.
info
(
"File {} has {} bytes, and will be truncated from {}."
,
file
.
getAbsolutePath
(),
file
.
length
(),
truncatedPosition
);
out
.
truncate
(
truncatedPosition
);
}
}
if
(!
autoRepair
)
{
out
.
close
();
throw
new
IOException
(
String
.
format
(
"%s is incomplete but does not allowed to be repaired..."
,
file
.
getAbsolutePath
()));
}
}
...
...
tsfile/src/main/java/org/apache/iotdb/tsfile/write/writer/TsFileIOWriter.java
浏览文件 @
742e6866
...
...
@@ -326,6 +326,7 @@ public class TsFileIOWriter {
* close the inputstream or file channel in force. This is just used for Testing.
*/
void
forceClose
()
throws
IOException
{
canWrite
=
false
;
out
.
close
();
}
...
...
tsfile/src/test/java/org/apache/iotdb/tsfile/write/writer/NativeRestorableIOWriterTest.java
浏览文件 @
742e6866
...
...
@@ -25,54 +25,62 @@ import static org.junit.Assert.assertNotEquals;
import
static
org
.
junit
.
Assert
.
assertTrue
;
import
java.io.File
;
import
java.io.FileWriter
;
import
java.io.IOException
;
import
java.nio.ByteBuffer
;
import
java.util.List
;
import
java.util.stream.Collectors
;
import
org.apache.iotdb.tsfile.common.conf.TSFileConfig
;
import
org.apache.iotdb.tsfile.common.conf.TSFileDescriptor
;
import
org.apache.iotdb.tsfile.encoding.decoder.Decoder
;
import
org.apache.iotdb.tsfile.file.MetaMarker
;
import
org.apache.iotdb.tsfile.file.footer.ChunkGroupFooter
;
import
org.apache.iotdb.tsfile.file.header.ChunkHeader
;
import
org.apache.iotdb.tsfile.file.header.PageHeader
;
import
org.apache.iotdb.tsfile.file.metadata.ChunkGroupMetaData
;
import
org.apache.iotdb.tsfile.file.metadata.ChunkMetaData
;
import
org.apache.iotdb.tsfile.file.metadata.TsDeviceMetadata
;
import
org.apache.iotdb.tsfile.file.metadata.TsDeviceMetadataIndex
;
import
org.apache.iotdb.tsfile.file.metadata.TsFileMetaData
;
import
org.apache.iotdb.tsfile.file.metadata.enums.CompressionType
;
import
org.apache.iotdb.tsfile.file.metadata.enums.TSDataType
;
import
org.apache.iotdb.tsfile.file.metadata.enums.TSEncoding
;
import
org.apache.iotdb.tsfile.file.metadata.statistics.FloatStatistics
;
import
org.apache.iotdb.tsfile.read.TsFileSequenceReader
;
import
org.apache.iotdb.tsfile.read.common.BatchData
;
import
org.apache.iotdb.tsfile.read.reader.page.PageReader
;
import
org.apache.iotdb.tsfile.write.TsFileWriter
;
import
org.apache.iotdb.tsfile.write.record.TSRecord
;
import
org.apache.iotdb.tsfile.write.record.datapoint.FloatDataPoint
;
import
org.apache.iotdb.tsfile.write.schema.MeasurementSchema
;
import
org.junit.After
;
import
org.junit.Before
;
import
org.junit.Test
;
public
class
NativeRestorableIOWriterTest
{
private
static
final
String
FILE_NAME
=
"test.ts"
;
@Test
public
void
testBadHeadMagic
()
throws
Exception
{
File
file
=
new
File
(
FILE_NAME
);
FileWriter
fWriter
=
new
FileWriter
(
file
);
fWriter
.
write
(
"Tsfile"
);
fWriter
.
close
();
try
{
NativeRestorableIOWriter
rWriter
=
new
NativeRestorableIOWriter
(
file
);
}
catch
(
IOException
e
)
{
assertTrue
(
e
.
getMessage
().
contains
(
"is not using TsFile format"
));
}
assertTrue
(
file
.
delete
());
}
@Test
public
void
testOnlyHeadMagic
()
throws
Exception
{
File
file
=
new
File
(
FILE_NAME
);
TsFileWriter
writer
=
new
TsFileWriter
(
file
);
writer
.
getIOWriter
().
forceClose
();
try
{
NativeRestorableIOWriter
rWriter
=
new
NativeRestorableIOWriter
(
file
,
false
);
writer
=
new
TsFileWriter
(
rWriter
);
writer
.
close
();
}
catch
(
IOException
e
)
{
assertTrue
(
e
.
getMessage
().
contains
(
"only has header"
));
}
NativeRestorableIOWriter
rWriter
=
new
NativeRestorableIOWriter
(
file
);
assertEquals
(
TsFileIOWriter
.
magicStringBytes
.
length
,
rWriter
.
getTruncatedPosition
());
writer
=
new
TsFileWriter
(
rWriter
);
writer
.
close
();
assertEquals
(
TsFileIOWriter
.
magicStringBytes
.
length
,
rWriter
.
getTruncatedPosition
());
assertTrue
(
file
.
delete
());
}
@Test
public
void
testOnlyFirstMask
()
throws
Exception
{
File
file
=
new
File
(
FILE_NAME
);
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录