Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
doujutun3207
flink
提交
af477563
F
flink
项目概览
doujutun3207
/
flink
与 Fork 源项目一致
从无法访问的项目Fork
通知
24
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
F
flink
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
af477563
编写于
9月 29, 2015
作者:
S
Stephan Ewen
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
[FLINK-2763] [runtime] Fix hash table spilling partition selection.
上级
16afb8ec
变更
3
隐藏空白更改
内联
并排
Showing
3 changed file
with
85 addition
and
3 deletion
+85
-3
flink-runtime/src/main/java/org/apache/flink/runtime/operators/hash/HashPartition.java
...rg/apache/flink/runtime/operators/hash/HashPartition.java
+14
-1
flink-runtime/src/main/java/org/apache/flink/runtime/operators/hash/MutableHashTable.java
...apache/flink/runtime/operators/hash/MutableHashTable.java
+2
-2
flink-runtime/src/test/java/org/apache/flink/runtime/operators/hash/HashTableTest.java
...rg/apache/flink/runtime/operators/hash/HashTableTest.java
+69
-0
未找到文件。
flink-runtime/src/main/java/org/apache/flink/runtime/operators/hash/HashPartition.java
浏览文件 @
af477563
...
...
@@ -198,6 +198,19 @@ public class HashPartition<BT, PT> extends AbstractPagedInputView implements See
public
final
boolean
isInMemory
()
{
return
this
.
buildSideChannel
==
null
;
}
/**
* Gets the number of memory segments used by this partition, which includes build side
* memory buffers and overflow memory segments.
*
* @return The number of occupied memory segments.
*/
public
int
getNumOccupiedMemorySegments
()
{
// either the number of memory segments, or one for spilling
final
int
numPartitionBuffers
=
this
.
partitionBuffers
!=
null
?
this
.
partitionBuffers
.
length
:
1
;
return
numPartitionBuffers
+
numOverflowSegments
;
}
public
int
getBuildSideBlockCount
()
{
return
this
.
partitionBuffers
==
null
?
this
.
buildSideWriteBuffer
.
getBlockCount
()
:
this
.
partitionBuffers
.
length
;
...
...
@@ -284,7 +297,7 @@ public class HashPartition<BT, PT> extends AbstractPagedInputView implements See
throw
new
RuntimeException
(
"Bug in Hybrid Hash Join: "
+
"Request to spill a partition that has already been spilled."
);
}
if
(
get
BuildSideBlockCount
()
+
this
.
numOverflowSegments
<
2
)
{
if
(
get
NumOccupiedMemorySegments
()
<
2
)
{
throw
new
RuntimeException
(
"Bug in Hybrid Hash Join: "
+
"Request to spill a partition with less than two buffers."
);
}
...
...
flink-runtime/src/main/java/org/apache/flink/runtime/operators/hash/MutableHashTable.java
浏览文件 @
af477563
...
...
@@ -1093,8 +1093,8 @@ public class MutableHashTable<BT, PT> implements MemorySegmentSource {
for
(
int
i
=
0
;
i
<
partitions
.
size
();
i
++)
{
HashPartition
<
BT
,
PT
>
p
=
partitions
.
get
(
i
);
if
(
p
.
isInMemory
()
&&
p
.
get
BuildSideBlockCount
()
>
largestNumBlocks
)
{
largestNumBlocks
=
p
.
get
BuildSideBlockCount
();
if
(
p
.
isInMemory
()
&&
p
.
get
NumOccupiedMemorySegments
()
>
largestNumBlocks
)
{
largestNumBlocks
=
p
.
get
NumOccupiedMemorySegments
();
largestPartNum
=
i
;
}
}
...
...
flink-runtime/src/test/java/org/apache/flink/runtime/operators/hash/HashTableTest.java
浏览文件 @
af477563
...
...
@@ -21,19 +21,23 @@ package org.apache.flink.runtime.operators.hash;
import
org.apache.flink.api.common.typeutils.TypeComparator
;
import
org.apache.flink.api.common.typeutils.TypePairComparator
;
import
org.apache.flink.api.common.typeutils.TypeSerializer
;
import
org.apache.flink.api.common.typeutils.base.ByteValueSerializer
;
import
org.apache.flink.api.common.typeutils.base.LongComparator
;
import
org.apache.flink.api.common.typeutils.base.LongSerializer
;
import
org.apache.flink.api.common.typeutils.base.array.BytePrimitiveArraySerializer
;
import
org.apache.flink.api.java.tuple.Tuple2
;
import
org.apache.flink.api.java.typeutils.runtime.TupleComparator
;
import
org.apache.flink.api.java.typeutils.runtime.TupleSerializer
;
import
org.apache.flink.api.java.typeutils.runtime.ValueComparator
;
import
org.apache.flink.core.memory.MemorySegment
;
import
org.apache.flink.core.memory.MemorySegmentFactory
;
import
org.apache.flink.runtime.io.disk.iomanager.IOManager
;
import
org.apache.flink.runtime.io.disk.iomanager.IOManagerAsync
;
import
org.apache.flink.types.ByteValue
;
import
org.apache.flink.util.MutableObjectIterator
;
import
org.junit.Test
;
import
org.mockito.Mockito
;
import
java.io.File
;
import
java.util.ArrayList
;
...
...
@@ -146,6 +150,47 @@ public class HashTableTest {
ioMan
.
shutdown
();
}
}
/**
* This tests the case where no additional partition buffers are used at the point when spilling
* is triggered, testing that overflow bucket buffers are taken into account when deciding which
* partition to spill.
*/
@Test
public
void
testSpillingFreesOnlyOverflowSegments
()
{
final
IOManager
ioMan
=
new
IOManagerAsync
();
final
TypeSerializer
<
ByteValue
>
serializer
=
ByteValueSerializer
.
INSTANCE
;
final
TypeComparator
<
ByteValue
>
buildComparator
=
new
ValueComparator
<>(
true
,
ByteValue
.
class
);
final
TypeComparator
<
ByteValue
>
probeComparator
=
new
ValueComparator
<>(
true
,
ByteValue
.
class
);
@SuppressWarnings
(
"unchecked"
)
final
TypePairComparator
<
ByteValue
,
ByteValue
>
pairComparator
=
Mockito
.
mock
(
TypePairComparator
.
class
);
try
{
final
int
pageSize
=
32
*
1024
;
final
int
numSegments
=
34
;
List
<
MemorySegment
>
memory
=
getMemory
(
numSegments
,
pageSize
);
MutableHashTable
<
ByteValue
,
ByteValue
>
table
=
new
MutableHashTable
<>(
serializer
,
serializer
,
buildComparator
,
probeComparator
,
pairComparator
,
memory
,
ioMan
,
1
,
false
);
table
.
open
(
new
ByteValueIterator
(
100000000
),
new
ByteValueIterator
(
1
));
table
.
close
();
checkNoTempFilesRemain
(
ioMan
);
}
catch
(
Exception
e
)
{
e
.
printStackTrace
();
fail
(
e
.
getMessage
());
}
finally
{
ioMan
.
shutdown
();
}
}
// ------------------------------------------------------------------------
// Utilities
...
...
@@ -219,4 +264,28 @@ public class HashTableTest {
}
}
}
private
static
class
ByteValueIterator
implements
MutableObjectIterator
<
ByteValue
>
{
private
final
long
numRecords
;
private
long
value
=
0
;
ByteValueIterator
(
long
numRecords
)
{
this
.
numRecords
=
numRecords
;
}
@Override
public
ByteValue
next
(
ByteValue
aLong
)
{
return
next
();
}
@Override
public
ByteValue
next
()
{
if
(
value
++
<
numRecords
)
{
return
new
ByteValue
((
byte
)
0
);
}
else
{
return
null
;
}
}
}
}
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录