Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
apache
Iotdb
提交
061a9787
I
Iotdb
项目概览
apache
/
Iotdb
12 个月 前同步成功
通知
25
Star
3344
Fork
916
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
DevOps
流水线
流水线任务
计划
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
I
Iotdb
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
DevOps
DevOps
流水线
流水线任务
计划
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
流水线任务
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
未验证
提交
061a9787
编写于
1月 20, 2022
作者:
周
周沛辰
提交者:
GitHub
1月 20, 2022
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
[IOTDB-2446]fix deleting bug in a compaction (#4916)
上级
5d09b8cc
变更
11
展开全部
隐藏空白更改
内联
并排
Showing
11 changed file
with
503 addition
and
50 deletion
+503
-50
integration/src/test/java/org/apache/iotdb/db/integration/IoTDBCompactionIT.java
...va/org/apache/iotdb/db/integration/IoTDBCompactionIT.java
+2
-2
integration/src/test/resources/logback.xml
integration/src/test/resources/logback.xml
+1
-1
server/src/main/java/org/apache/iotdb/db/engine/compaction/CompactionUtils.java
...rg/apache/iotdb/db/engine/compaction/CompactionUtils.java
+3
-3
server/src/main/java/org/apache/iotdb/db/engine/compaction/cross/AbstractCrossSpaceCompactionRecoverTask.java
...action/cross/AbstractCrossSpaceCompactionRecoverTask.java
+0
-31
server/src/main/java/org/apache/iotdb/db/engine/compaction/inner/utils/InnerSpaceCompactionUtils.java
...ine/compaction/inner/utils/InnerSpaceCompactionUtils.java
+0
-1
server/src/main/java/org/apache/iotdb/db/engine/storagegroup/TsFileResource.java
...g/apache/iotdb/db/engine/storagegroup/TsFileResource.java
+10
-4
server/src/main/java/org/apache/iotdb/db/engine/storagegroup/VirtualStorageGroupProcessor.java
.../db/engine/storagegroup/VirtualStorageGroupProcessor.java
+4
-1
server/src/test/java/org/apache/iotdb/db/engine/compaction/AbstractCompactionTest.java
...he/iotdb/db/engine/compaction/AbstractCompactionTest.java
+6
-5
server/src/test/java/org/apache/iotdb/db/engine/compaction/cross/RewriteCrossSpaceCompactionTest.java
...ine/compaction/cross/RewriteCrossSpaceCompactionTest.java
+475
-0
server/src/test/java/org/apache/iotdb/db/engine/compaction/task/FakedCrossSpaceCompactionTask.java
...engine/compaction/task/FakedCrossSpaceCompactionTask.java
+1
-1
server/src/test/resources/logback.xml
server/src/test/resources/logback.xml
+1
-1
未找到文件。
integration/src/test/java/org/apache/iotdb/db/integration/IoTDB
Merge
IT.java
→
integration/src/test/java/org/apache/iotdb/db/integration/IoTDB
Compaction
IT.java
浏览文件 @
061a9787
...
...
@@ -42,9 +42,9 @@ import static org.junit.Assert.assertEquals;
import
static
org
.
junit
.
Assert
.
fail
;
@Category
({
LocalStandaloneTest
.
class
,
ClusterTest
.
class
})
public
class
IoTDB
Merge
IT
{
public
class
IoTDB
Compaction
IT
{
private
static
final
Logger
logger
=
LoggerFactory
.
getLogger
(
IoTDB
Merge
IT
.
class
);
private
static
final
Logger
logger
=
LoggerFactory
.
getLogger
(
IoTDB
Compaction
IT
.
class
);
private
long
prevPartitionInterval
;
@Before
...
...
integration/src/test/resources/logback.xml
浏览文件 @
061a9787
...
...
@@ -45,7 +45,7 @@
<logger
name=
"org.eclipse.jetty.util.thread.QueuedThreadPool"
level=
"INFO"
/>
<logger
name=
"org.apache.iotdb.db.service.metrics.MetricsService"
level=
"INFO"
/>
<logger
name=
"org.apache.iotdb.db.engine.flush.FlushManager"
level=
"INFO"
/>
<logger
name=
"org.apache.iotdb.db.integration.IoTDB
Merge
IT"
level=
"INFO"
/>
<logger
name=
"org.apache.iotdb.db.integration.IoTDB
Compaction
IT"
level=
"INFO"
/>
<logger
name=
"org.apache.iotdb.db.service.RegisterManager"
level=
"INFO"
/>
<logger
name=
"org.apache.iotdb.db.service.IoTDB"
level=
"WARN"
/>
<logger
name=
"org.apache.iotdb.db.service.RPCService"
level=
"INFO"
/>
...
...
server/src/main/java/org/apache/iotdb/db/engine/compaction/CompactionUtils.java
浏览文件 @
061a9787
...
...
@@ -284,7 +284,6 @@ public class CompactionUtils {
}
else
{
fileSuffix
=
IoTDBConstant
.
CROSS_COMPACTION_TMP_FILE_SUFFIX
;
}
// checkAndUpdateTargetFileResources(targetResources, fileSuffix);
for
(
TsFileResource
targetResource
:
targetResources
)
{
moveOneTargetFile
(
targetResource
,
fileSuffix
,
fullStorageGroupName
);
}
...
...
@@ -304,7 +303,9 @@ public class CompactionUtils {
File
newFile
=
new
File
(
targetResource
.
getTsFilePath
().
replace
(
tmpFileSuffix
,
TsFileConstant
.
TSFILE_SUFFIX
));
FSFactoryProducer
.
getFSFactory
().
moveFile
(
targetResource
.
getTsFile
(),
newFile
);
if
(!
newFile
.
exists
())
{
FSFactoryProducer
.
getFSFactory
().
moveFile
(
targetResource
.
getTsFile
(),
newFile
);
}
// serialize xxx.tsfile.resource
targetResource
.
setFile
(
newFile
);
...
...
@@ -374,7 +375,6 @@ public class CompactionUtils {
ModificationFile
.
getCompactionMods
(
sourceFile
);
Collection
<
Modification
>
newModification
=
compactionModificationFile
.
getModifications
();
compactionModificationFile
.
close
();
sourceFile
.
resetModFile
();
// write the new modifications to its old modification file
try
(
ModificationFile
oldModificationFile
=
sourceFile
.
getModFile
())
{
for
(
Modification
modification
:
newModification
)
{
...
...
server/src/main/java/org/apache/iotdb/db/engine/compaction/cross/AbstractCrossSpaceCompactionRecoverTask.java
已删除
100644 → 0
浏览文件 @
5d09b8cc
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
package
org.apache.iotdb.db.engine.compaction.cross
;
import
java.util.concurrent.atomic.AtomicInteger
;
public
abstract
class
AbstractCrossSpaceCompactionRecoverTask
extends
AbstractCrossSpaceCompactionTask
{
public
AbstractCrossSpaceCompactionRecoverTask
(
String
fullStorageGroupName
,
long
timePartition
,
AtomicInteger
currentTaskNum
)
{
super
(
fullStorageGroupName
,
timePartition
,
currentTaskNum
);
}
}
server/src/main/java/org/apache/iotdb/db/engine/compaction/inner/utils/InnerSpaceCompactionUtils.java
浏览文件 @
061a9787
...
...
@@ -180,7 +180,6 @@ public class InnerSpaceCompactionUtils {
ModificationFile
.
getCompactionMods
(
sourceFile
);
Collection
<
Modification
>
newModification
=
compactionModificationFile
.
getModifications
();
compactionModificationFile
.
close
();
sourceFile
.
resetModFile
();
// write the new modifications to its old modification file
try
(
ModificationFile
oldModificationFile
=
sourceFile
.
getModFile
())
{
for
(
Modification
modification
:
newModification
)
{
...
...
server/src/main/java/org/apache/iotdb/db/engine/storagegroup/TsFileResource.java
浏览文件 @
061a9787
...
...
@@ -97,6 +97,8 @@ public class TsFileResource {
private
ModificationFile
modFile
;
private
ModificationFile
compactionModFile
;
protected
volatile
boolean
closed
=
false
;
private
volatile
boolean
deleted
=
false
;
volatile
boolean
isMerging
=
false
;
...
...
@@ -357,14 +359,14 @@ public class TsFileResource {
}
public
ModificationFile
getCompactionModFile
()
{
if
(
m
odFile
==
null
)
{
if
(
compactionM
odFile
==
null
)
{
synchronized
(
this
)
{
if
(
m
odFile
==
null
)
{
m
odFile
=
ModificationFile
.
getCompactionMods
(
this
);
if
(
compactionM
odFile
==
null
)
{
compactionM
odFile
=
ModificationFile
.
getCompactionMods
(
this
);
}
}
}
return
m
odFile
;
return
compactionM
odFile
;
}
public
void
resetModFile
()
{
...
...
@@ -442,6 +444,10 @@ public class TsFileResource {
modFile
.
close
();
modFile
=
null
;
}
if
(
compactionModFile
!=
null
)
{
compactionModFile
.
close
();
compactionModFile
=
null
;
}
processor
=
null
;
pathToChunkMetadataListMap
=
null
;
pathToReadOnlyMemChunkMap
=
null
;
...
...
server/src/main/java/org/apache/iotdb/db/engine/storagegroup/VirtualStorageGroupProcessor.java
浏览文件 @
061a9787
...
...
@@ -2006,10 +2006,13 @@ public class VirtualStorageGroupProcessor {
// we have to set modification offset to MAX_VALUE, as the offset of source chunk may
// change after compaction
deletion
.
setFileOffset
(
Long
.
MAX_VALUE
);
// write deletion into modification file
// write deletion into
compaction
modification file
tsFileResource
.
getCompactionModFile
().
write
(
deletion
);
// write deletion into modification file to enable query during compaction
tsFileResource
.
getModFile
().
write
(
deletion
);
// remember to close mod file
tsFileResource
.
getCompactionModFile
().
close
();
tsFileResource
.
getModFile
().
close
();
}
else
{
deletion
.
setFileOffset
(
tsFileResource
.
getTsFileSize
());
// write deletion into modification file
...
...
server/src/test/java/org/apache/iotdb/db/engine/compaction/AbstractCompactionTest.java
浏览文件 @
061a9787
...
...
@@ -191,15 +191,16 @@ public class AbstractCompactionTest {
TsFileResource
resource
=
new
TsFileResource
(
file
);
int
deviceStartindex
=
0
;
if
(
isAlign
)
{
deviceStartindex
=
TsFileGeneratorUtils
.
getAlignDeviceOffset
();
for
(
int
i
=
deviceStartindex
;
i
<
deviceStartindex
+
deviceNum
;
i
++)
{
resource
.
updateStartTime
(
COMPACTION_TEST_SG
+
PATH_SEPARATOR
+
"d"
+
i
,
startTime
);
resource
.
updateEndTime
(
COMPACTION_TEST_SG
+
PATH_SEPARATOR
+
"d"
+
i
,
endTime
);
}
}
else
{
for
(
int
i
=
deviceStartindex
;
i
<
deviceStartindex
+
deviceNum
;
i
++)
{
resource
.
updateStartTime
(
COMPACTION_TEST_SG
+
PATH_SEPARATOR
+
"d"
+
i
,
startTime
);
resource
.
updateEndTime
(
COMPACTION_TEST_SG
+
PATH_SEPARATOR
+
"d"
+
i
,
endTime
);
}
deviceStartindex
=
TsFileGeneratorUtils
.
getAlignDeviceOffset
();
}
for
(
int
i
=
deviceStartindex
;
i
<
deviceStartindex
+
deviceNum
;
i
++)
{
resource
.
updateStartTime
(
COMPACTION_TEST_SG
+
PATH_SEPARATOR
+
"d"
+
i
,
startTime
);
resource
.
updateEndTime
(
COMPACTION_TEST_SG
+
PATH_SEPARATOR
+
"d"
+
i
,
endTime
);
}
resource
.
updatePlanIndexes
(
fileVersion
);
resource
.
setClosed
(
true
);
...
...
server/src/test/java/org/apache/iotdb/db/engine/compaction/cross/RewriteCrossSpaceCompactionTest.java
浏览文件 @
061a9787
此差异已折叠。
点击以展开。
server/src/test/java/org/apache/iotdb/db/engine/compaction/task/FakedCrossSpaceCompactionTask.java
浏览文件 @
061a9787
...
...
@@ -58,7 +58,7 @@ public class FakedCrossSpaceCompactionTask extends RewriteCrossSpaceCompactionTa
((
FakedTsFileResource
)
resource
)
.
setTsFileSize
(
resource
.
getTsFileSize
()
+
avgSizeAddToSeqFile
);
}
selectedSeqTsFileResourceList
.
clear
();
selectedUnSeqTsFileResourceList
.
clear
();
// unSeqTsFileResourceList.clear();
}
}
server/src/test/resources/logback.xml
浏览文件 @
061a9787
...
...
@@ -45,7 +45,7 @@
<logger
name=
"org.eclipse.jetty.util.thread.QueuedThreadPool"
level=
"INFO"
/>
<logger
name=
"org.apache.iotdb.db.service.metrics.MetricsService"
level=
"INFO"
/>
<logger
name=
"org.apache.iotdb.db.engine.flush.FlushManager"
level=
"INFO"
/>
<logger
name=
"org.apache.iotdb.db.integration.IoTDB
Merge
IT"
level=
"INFO"
/>
<logger
name=
"org.apache.iotdb.db.integration.IoTDB
Compaction
IT"
level=
"INFO"
/>
<logger
name=
"org.apache.iotdb.db.service.RegisterManager"
level=
"INFO"
/>
<logger
name=
"org.apache.iotdb.db.service.IoTDB"
level=
"WARN"
/>
<logger
name=
"org.apache.iotdb.db.service.RPCService"
level=
"INFO"
/>
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录