Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
apache
Iotdb
提交
971ec2b6
I
Iotdb
项目概览
apache
/
Iotdb
大约 1 年 前同步成功
通知
25
Star
3344
Fork
916
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
DevOps
流水线
流水线任务
计划
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
I
Iotdb
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
DevOps
DevOps
流水线
流水线任务
计划
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
流水线任务
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
971ec2b6
编写于
3月 06, 2019
作者:
J
Jiang Tian
提交者:
Xiangdong Huang
3月 06, 2019
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
Add wal checking tool (#85)
* add wal checker * fix show timeseries IT test
上级
cd8d5fdb
变更
11
显示空白变更内容
内联
并排
Showing
11 changed file
with
331 addition
and
39 deletion
+331
-39
iotdb/src/main/java/org/apache/iotdb/db/exception/SysCheckException.java
...java/org/apache/iotdb/db/exception/SysCheckException.java
+38
-0
iotdb/src/main/java/org/apache/iotdb/db/tools/WalChecker.java
...b/src/main/java/org/apache/iotdb/db/tools/WalChecker.java
+118
-0
iotdb/src/main/java/org/apache/iotdb/db/writelog/io/ILogReader.java
...main/java/org/apache/iotdb/db/writelog/io/ILogReader.java
+6
-1
iotdb/src/main/java/org/apache/iotdb/db/writelog/io/RAFLogReader.java
...in/java/org/apache/iotdb/db/writelog/io/RAFLogReader.java
+18
-27
iotdb/src/main/java/org/apache/iotdb/db/writelog/recover/ExclusiveLogRecoverPerformer.java
...tdb/db/writelog/recover/ExclusiveLogRecoverPerformer.java
+13
-5
iotdb/src/main/java/org/apache/iotdb/db/writelog/transfer/PhysicalPlanCodec.java
.../apache/iotdb/db/writelog/transfer/PhysicalPlanCodec.java
+2
-2
iotdb/src/main/java/org/apache/iotdb/db/writelog/transfer/PhysicalPlanLogTransfer.java
...e/iotdb/db/writelog/transfer/PhysicalPlanLogTransfer.java
+1
-1
iotdb/src/test/java/org/apache/iotdb/db/tools/WalCheckerTest.java
...c/test/java/org/apache/iotdb/db/tools/WalCheckerTest.java
+133
-0
iotdb/src/test/java/org/apache/iotdb/db/writelog/PerformanceTest.java
...st/java/org/apache/iotdb/db/writelog/PerformanceTest.java
+1
-1
iotdb/src/test/java/org/apache/iotdb/db/writelog/io/LogWriterReaderTest.java
.../org/apache/iotdb/db/writelog/io/LogWriterReaderTest.java
+1
-1
tsfile/pom.xml
tsfile/pom.xml
+0
-1
未找到文件。
iotdb/src/main/java/org/apache/iotdb/db/exception/SysCheckException.java
0 → 100644
浏览文件 @
971ec2b6
/**
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
package
org.apache.iotdb.db.exception
;
public
class
SysCheckException
extends
Exception
{
public
SysCheckException
()
{
}
public
SysCheckException
(
String
message
)
{
super
(
message
);
}
public
SysCheckException
(
String
message
,
Throwable
cause
)
{
super
(
message
,
cause
);
}
public
SysCheckException
(
Throwable
cause
)
{
super
(
cause
);
}
}
iotdb/src/main/java/org/apache/iotdb/db/tools/WalChecker.java
0 → 100644
浏览文件 @
971ec2b6
/**
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
package
org.apache.iotdb.db.tools
;
import
static
org
.
apache
.
iotdb
.
db
.
writelog
.
node
.
ExclusiveWriteLogNode
.
WAL_FILE_NAME
;
import
java.io.File
;
import
java.io.IOException
;
import
java.util.ArrayList
;
import
java.util.Collections
;
import
java.util.List
;
import
org.apache.iotdb.db.exception.SysCheckException
;
import
org.apache.iotdb.db.writelog.io.RAFLogReader
;
import
org.slf4j.Logger
;
import
org.slf4j.LoggerFactory
;
/**
* WalChecker verifies that whether all write ahead logs in the WAL folder are recognizable.
*/
public
class
WalChecker
{
private
static
final
Logger
LOGGER
=
LoggerFactory
.
getLogger
(
WalChecker
.
class
);
/**
* the root dir of wals, which should have wal directories of storage groups as its children.
*/
private
String
walFolder
;
public
WalChecker
(
String
walFolder
)
{
this
.
walFolder
=
walFolder
;
}
/**
* check the root wal dir and find the damaged files
* @return a list of damaged files.
* @throws SysCheckException if the root wal dir does not exist.
*/
public
List
<
File
>
doCheck
()
throws
SysCheckException
{
File
walFolderFile
=
new
File
(
walFolder
);
if
(!
walFolderFile
.
exists
()
||
!
walFolderFile
.
isDirectory
())
{
throw
new
SysCheckException
(
String
.
format
(
"%s is not a directory"
,
walFolder
));
}
File
[]
storageWalFolders
=
walFolderFile
.
listFiles
();
if
(
storageWalFolders
==
null
||
storageWalFolders
.
length
==
0
)
{
LOGGER
.
info
(
"No sub-directories under the given directory, check ends"
);
return
Collections
.
emptyList
();
}
List
<
File
>
failedFiles
=
new
ArrayList
<>();
for
(
int
dirIndex
=
0
;
dirIndex
<
storageWalFolders
.
length
;
dirIndex
++)
{
File
storageWalFolder
=
storageWalFolders
[
dirIndex
];
LOGGER
.
debug
(
"Checking the No.{} directory {}"
,
dirIndex
,
storageWalFolder
.
getName
());
File
walFile
=
new
File
(
storageWalFolder
,
WAL_FILE_NAME
);
if
(!
walFile
.
exists
())
{
LOGGER
.
debug
(
"No wal file in this dir, skipping"
);
continue
;
}
RAFLogReader
logReader
=
null
;
try
{
logReader
=
new
RAFLogReader
(
walFile
);
while
(
logReader
.
hasNext
())
{
logReader
.
next
();
}
}
catch
(
IOException
e
)
{
failedFiles
.
add
(
walFile
);
LOGGER
.
error
(
"{} fails the check because"
,
walFile
.
getAbsoluteFile
(),
e
);
}
finally
{
if
(
logReader
!=
null
)
{
logReader
.
close
();
}
}
}
return
failedFiles
;
}
// a temporary method which should be in the integrated self-check module in the future
public
static
void
report
(
List
<
File
>
failedFiles
)
{
if
(
failedFiles
.
isEmpty
())
{
LOGGER
.
info
(
"Check finished. There is no damaged file"
);
}
else
{
LOGGER
.
error
(
"There are {} failed files. They are {}"
,
failedFiles
.
size
(),
failedFiles
);
}
}
/**
*
* @param args walRootDirectory
*/
public
static
void
main
(
String
[]
args
)
throws
SysCheckException
{
if
(
args
.
length
<
1
)
{
LOGGER
.
error
(
"No enough args: require the walRootDirectory"
);
return
;
}
WalChecker
checker
=
new
WalChecker
(
args
[
0
]);
List
<
File
>
files
=
checker
.
doCheck
();
report
(
files
);
}
}
iotdb/src/main/java/org/apache/iotdb/db/writelog/io/ILogReader.java
浏览文件 @
971ec2b6
...
...
@@ -20,12 +20,17 @@ package org.apache.iotdb.db.writelog.io;
import
java.io.File
;
import
java.io.FileNotFoundException
;
import
java.io.IOException
;
import
java.util.Iterator
;
import
org.apache.iotdb.db.qp.physical.PhysicalPlan
;
public
interface
ILogReader
extends
Iterator
<
PhysicalPlan
>
{
public
interface
ILogReader
{
void
open
(
File
file
)
throws
FileNotFoundException
;
void
close
();
boolean
hasNext
()
throws
IOException
;
PhysicalPlan
next
()
throws
IOException
;
}
iotdb/src/main/java/org/apache/iotdb/db/writelog/io/RAFLogReader.java
浏览文件 @
971ec2b6
...
...
@@ -48,19 +48,15 @@ public class RAFLogReader implements ILogReader {
}
@Override
public
boolean
hasNext
()
{
public
boolean
hasNext
()
throws
IOException
{
if
(
planBuffer
!=
null
)
{
return
true
;
}
try
{
if
(
logRaf
.
getFilePointer
()
+
12
>
logRaf
.
length
())
{
return
false
;
}
}
catch
(
IOException
e
)
{
logger
.
error
(
"Cannot read from log file {}"
,
filepath
,
e
);
return
false
;
}
try
{
int
logSize
=
logRaf
.
readInt
();
if
(
logSize
>
bufferSize
)
{
bufferSize
=
logSize
;
...
...
@@ -71,19 +67,14 @@ public class RAFLogReader implements ILogReader {
checkSummer
.
reset
();
checkSummer
.
update
(
buffer
,
0
,
logSize
);
if
(
checkSummer
.
getValue
()
!=
checkSum
)
{
return
false
;
throw
new
IOException
(
"The check sum is incorrect!"
)
;
}
PhysicalPlan
plan
=
PhysicalPlanLogTransfer
.
logToOperator
(
buffer
);
planBuffer
=
plan
;
planBuffer
=
PhysicalPlanLogTransfer
.
logToOperator
(
buffer
);
return
true
;
}
catch
(
IOException
e
)
{
logger
.
error
(
"Cannot read log file {}"
,
filepath
,
e
);
return
false
;
}
}
@Override
public
PhysicalPlan
next
()
{
public
PhysicalPlan
next
()
throws
IOException
{
if
(!
hasNext
()){
throw
new
NoSuchElementException
();
}
...
...
iotdb/src/main/java/org/apache/iotdb/db/writelog/recover/ExclusiveLogRecoverPerformer.java
浏览文件 @
971ec2b6
...
...
@@ -259,15 +259,15 @@ public class ExclusiveLogRecoverPerformer implements RecoverPerformer {
replayLog
();
}
private
int
replayLogFile
(
File
logFile
)
throws
RecoverException
{
private
int
replayLogFile
(
File
logFile
)
throws
RecoverException
,
IOException
{
int
failedCnt
=
0
;
if
(
logFile
.
exists
())
{
try
{
rafLogReader
.
open
(
logFile
);
}
catch
(
FileNotFoundException
e
)
{
logger
.
error
(
"Log node {} cannot read old log file, because
{}
"
,
writeLogNode
.
getIdentifier
(),
e
.
getMessage
()
);
.
error
(
"Log node {} cannot read old log file, because "
,
writeLogNode
.
getIdentifier
(),
e
);
throw
new
RecoverException
(
"Cannot read old log file, recovery aborted."
);
}
while
(
rafLogReader
.
hasNext
())
{
...
...
@@ -294,11 +294,19 @@ public class ExclusiveLogRecoverPerformer implements RecoverPerformer {
File
oldLogFile
=
new
File
(
writeLogNode
.
getLogDirectory
()
+
File
.
separator
+
ExclusiveWriteLogNode
.
WAL_FILE_NAME
+
ExclusiveWriteLogNode
.
OLD_SUFFIX
);
try
{
failedEntryCnt
+=
replayLogFile
(
oldLogFile
);
}
catch
(
IOException
e
)
{
throw
new
RecoverException
(
e
);
}
// then replay new log
File
newLogFile
=
new
File
(
writeLogNode
.
getLogDirectory
()
+
File
.
separator
+
ExclusiveWriteLogNode
.
WAL_FILE_NAME
);
try
{
failedEntryCnt
+=
replayLogFile
(
newLogFile
);
}
catch
(
IOException
e
)
{
throw
new
RecoverException
(
e
);
}
// TODO : do we need to proceed if there are failed logs ?
if
(
failedEntryCnt
>
0
)
{
throw
new
RecoverException
(
...
...
iotdb/src/main/java/org/apache/iotdb/db/writelog/transfer/PhysicalPlanCodec.java
浏览文件 @
971ec2b6
...
...
@@ -57,9 +57,9 @@ public enum PhysicalPlanCodec {
this
.
codec
=
codec
;
}
public
static
PhysicalPlanCodec
fromOpcode
(
int
opcode
)
{
public
static
PhysicalPlanCodec
fromOpcode
(
int
opcode
)
throws
IOException
{
if
(!
codecMap
.
containsKey
(
opcode
))
{
throw
new
UnsupportedOperation
Exception
(
throw
new
IO
Exception
(
"SystemLogOperator ["
+
opcode
+
"] is not supported. "
);
}
return
codecMap
.
get
(
opcode
);
...
...
iotdb/src/main/java/org/apache/iotdb/db/writelog/transfer/PhysicalPlanLogTransfer.java
浏览文件 @
971ec2b6
...
...
@@ -27,7 +27,7 @@ public class PhysicalPlanLogTransfer {
private
PhysicalPlanLogTransfer
(){}
public
static
byte
[]
operatorToLog
(
PhysicalPlan
plan
)
throws
WALOverSized
Exception
{
public
static
byte
[]
operatorToLog
(
PhysicalPlan
plan
)
throws
IO
Exception
{
Codec
<
PhysicalPlan
>
codec
;
switch
(
plan
.
getOperatorType
())
{
case
INSERT:
...
...
iotdb/src/test/java/org/apache/iotdb/db/tools/WalCheckerTest.java
0 → 100644
浏览文件 @
971ec2b6
/**
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
package
org.apache.iotdb.db.tools
;
import
static
org
.
apache
.
iotdb
.
db
.
writelog
.
node
.
ExclusiveWriteLogNode
.
WAL_FILE_NAME
;
import
static
org
.
junit
.
Assert
.
assertEquals
;
import
static
org
.
junit
.
Assert
.
assertTrue
;
import
java.io.File
;
import
java.io.IOException
;
import
java.util.ArrayList
;
import
java.util.Arrays
;
import
java.util.List
;
import
org.apache.commons.io.FileUtils
;
import
org.apache.iotdb.db.exception.SysCheckException
;
import
org.apache.iotdb.db.qp.physical.crud.InsertPlan
;
import
org.apache.iotdb.db.writelog.io.LogWriter
;
import
org.apache.iotdb.db.writelog.transfer.PhysicalPlanLogTransfer
;
import
org.junit.Test
;
public
class
WalCheckerTest
{
@Test
public
void
testNoDir
()
{
WalChecker
checker
=
new
WalChecker
(
"no such dir"
);
boolean
caught
=
false
;
try
{
checker
.
doCheck
();
}
catch
(
SysCheckException
e
)
{
caught
=
true
;
}
assertTrue
(
caught
);
}
@Test
public
void
testEmpty
()
throws
IOException
,
SysCheckException
{
File
tempRoot
=
new
File
(
"root"
);
tempRoot
.
mkdir
();
try
{
WalChecker
checker
=
new
WalChecker
(
tempRoot
.
getAbsolutePath
());
assertTrue
(
checker
.
doCheck
().
isEmpty
());
}
finally
{
FileUtils
.
deleteDirectory
(
tempRoot
);
}
}
@Test
public
void
testNormalCheck
()
throws
IOException
,
SysCheckException
{
File
tempRoot
=
new
File
(
"root"
);
tempRoot
.
mkdir
();
try
{
for
(
int
i
=
0
;
i
<
5
;
i
++)
{
File
subDir
=
new
File
(
tempRoot
,
"storage_group"
+
i
);
subDir
.
mkdir
();
LogWriter
logWriter
=
new
LogWriter
(
subDir
.
getPath
()
+
File
.
separator
+
WAL_FILE_NAME
);
List
<
byte
[]>
binaryPlans
=
new
ArrayList
<>();
String
deviceId
=
"device1"
;
List
<
String
>
measurements
=
Arrays
.
asList
(
"s1"
,
"s2"
,
"s3"
);
List
<
String
>
values
=
Arrays
.
asList
(
"5"
,
"6"
,
"7"
);
for
(
int
j
=
0
;
j
<
10
;
j
++)
{
binaryPlans
.
add
(
PhysicalPlanLogTransfer
.
operatorToLog
(
new
InsertPlan
(
deviceId
,
j
,
measurements
,
values
)));
}
logWriter
.
write
(
binaryPlans
);
logWriter
.
force
();
logWriter
.
close
();
}
WalChecker
checker
=
new
WalChecker
(
tempRoot
.
getAbsolutePath
());
assertTrue
(
checker
.
doCheck
().
isEmpty
());
}
finally
{
FileUtils
.
deleteDirectory
(
tempRoot
);
}
}
@Test
public
void
testAbnormalCheck
()
throws
IOException
,
SysCheckException
{
File
tempRoot
=
new
File
(
"root"
);
tempRoot
.
mkdir
();
try
{
for
(
int
i
=
0
;
i
<
5
;
i
++)
{
File
subDir
=
new
File
(
tempRoot
,
"storage_group"
+
i
);
subDir
.
mkdir
();
LogWriter
logWriter
=
new
LogWriter
(
subDir
.
getPath
()
+
File
.
separator
+
WAL_FILE_NAME
);
List
<
byte
[]>
binaryPlans
=
new
ArrayList
<>();
String
deviceId
=
"device1"
;
List
<
String
>
measurements
=
Arrays
.
asList
(
"s1"
,
"s2"
,
"s3"
);
List
<
String
>
values
=
Arrays
.
asList
(
"5"
,
"6"
,
"7"
);
for
(
int
j
=
0
;
j
<
10
;
j
++)
{
binaryPlans
.
add
(
PhysicalPlanLogTransfer
.
operatorToLog
(
new
InsertPlan
(
deviceId
,
j
,
measurements
,
values
)));
}
if
(
i
>
2
)
{
binaryPlans
.
add
(
"not a wal"
.
getBytes
());
}
logWriter
.
write
(
binaryPlans
);
logWriter
.
force
();
logWriter
.
close
();
}
WalChecker
checker
=
new
WalChecker
(
tempRoot
.
getAbsolutePath
());
assertEquals
(
2
,
checker
.
doCheck
().
size
());
}
finally
{
FileUtils
.
deleteDirectory
(
tempRoot
);
}
}
}
iotdb/src/test/java/org/apache/iotdb/db/writelog/PerformanceTest.java
浏览文件 @
971ec2b6
...
...
@@ -216,7 +216,7 @@ public class PerformanceTest {
}
@Test
public
void
SQLEncodingComparisonTest
()
throws
WALOverSized
Exception
{
public
void
SQLEncodingComparisonTest
()
throws
IO
Exception
{
String
sql
=
"INSERT INTO root.logTestDevice(time,s1,s2,s3,s4) "
+
"VALUES (100,1.0,15,\"str\",false)"
;
InsertPlan
bwInsertPlan
=
new
InsertPlan
(
1
,
"root.logTestDevice"
,
100
,
...
...
iotdb/src/test/java/org/apache/iotdb/db/writelog/io/LogWriterReaderTest.java
浏览文件 @
971ec2b6
...
...
@@ -42,7 +42,7 @@ public class LogWriterReaderTest {
List
<
PhysicalPlan
>
plans
=
new
ArrayList
<>();
@Before
public
void
prepare
()
throws
WALOverSized
Exception
{
public
void
prepare
()
throws
IO
Exception
{
if
(
new
File
(
filePath
).
exists
())
{
new
File
(
filePath
).
delete
();
}
...
...
tsfile/pom.xml
浏览文件 @
971ec2b6
...
...
@@ -113,7 +113,6 @@
</execution>
</executions>
</plugin>
</plugins>
</build>
<profiles>
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录