Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
doujutun3207
flink
提交
acd15d25
F
flink
项目概览
doujutun3207
/
flink
与 Fork 源项目一致
从无法访问的项目Fork
通知
24
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
F
flink
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
acd15d25
编写于
7月 26, 2019
作者:
Z
Zhijiang
提交者:
Stephan Ewen
7月 29, 2019
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
[FLINK-13245][network] Make subpartition consumption notification independant
上级
653ba1a1
变更
2
隐藏空白更改
内联
并排
Showing
2 changed file
with
67 addition
and
16 deletion
+67
-16
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/ReleaseOnConsumptionResultPartition.java
...etwork/partition/ReleaseOnConsumptionResultPartition.java
+30
-12
flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/ReleaseOnConsumptionResultPartitionTest.java
...rk/partition/ReleaseOnConsumptionResultPartitionTest.java
+37
-4
未找到文件。
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/ReleaseOnConsumptionResultPartition.java
浏览文件 @
acd15d25
...
...
@@ -22,7 +22,6 @@ import org.apache.flink.runtime.io.network.buffer.BufferPoolOwner;
import
org.apache.flink.util.function.FunctionWithException
;
import
java.io.IOException
;
import
java.util.concurrent.atomic.AtomicInteger
;
import
static
org
.
apache
.
flink
.
util
.
Preconditions
.
checkState
;
...
...
@@ -31,11 +30,18 @@ import static org.apache.flink.util.Preconditions.checkState;
*/
public
class
ReleaseOnConsumptionResultPartition
extends
ResultPartition
{
private
static
final
Object
lock
=
new
Object
();
/**
* A flag for each subpartition indicating whether it was already consumed or not.
*/
private
final
boolean
[]
consumedSubpartitions
;
/**
* The total number of references to subpartitions of this result. The result partition can be
* safely released, iff the reference count is zero.
*/
private
final
AtomicInteger
pendingReferences
=
new
AtomicInteger
()
;
private
int
numUnconsumedSubpartitions
;
ReleaseOnConsumptionResultPartition
(
String
owningTaskName
,
...
...
@@ -47,12 +53,13 @@ public class ReleaseOnConsumptionResultPartition extends ResultPartition {
FunctionWithException
<
BufferPoolOwner
,
BufferPool
,
IOException
>
bufferPoolFactory
)
{
super
(
owningTaskName
,
partitionId
,
partitionType
,
subpartitions
,
numTargetKeyGroups
,
partitionManager
,
bufferPoolFactory
);
pendingReferences
.
set
(
subpartitions
.
length
);
this
.
consumedSubpartitions
=
new
boolean
[
subpartitions
.
length
];
this
.
numUnconsumedSubpartitions
=
subpartitions
.
length
;
}
@Override
public
ResultSubpartitionView
createSubpartitionView
(
int
index
,
BufferAvailabilityListener
availabilityListener
)
throws
IOException
{
checkState
(
pendingReferences
.
get
()
>
0
,
"Partition not pinned."
);
checkState
(
numUnconsumedSubpartitions
>
0
,
"Partition not pinned."
);
return
super
.
createSubpartitionView
(
index
,
availabilityListener
);
}
...
...
@@ -63,22 +70,33 @@ public class ReleaseOnConsumptionResultPartition extends ResultPartition {
return
;
}
int
refCnt
=
pendingReferences
.
decrementAndGet
()
;
final
int
remainingUnconsumed
;
if
(
refCnt
==
0
)
{
partitionManager
.
onConsumedPartition
(
this
);
}
else
if
(
refCnt
<
0
)
{
throw
new
IllegalStateException
(
"All references released."
);
// we synchronize only the bookkeeping section, to avoid holding the lock during any
// calls into other components
synchronized
(
lock
)
{
if
(
consumedSubpartitions
[
subpartitionIndex
])
{
// repeated call - ignore
return
;
}
consumedSubpartitions
[
subpartitionIndex
]
=
true
;
remainingUnconsumed
=
(--
numUnconsumedSubpartitions
);
}
LOG
.
debug
(
"{}: Received release notification for subpartition {}."
,
this
,
subpartitionIndex
);
LOG
.
debug
(
"{}: Received consumed notification for subpartition {}."
,
this
,
subpartitionIndex
);
if
(
remainingUnconsumed
==
0
)
{
partitionManager
.
onConsumedPartition
(
this
);
}
else
if
(
remainingUnconsumed
<
0
)
{
throw
new
IllegalStateException
(
"Received consume notification even though all subpartitions are already consumed."
);
}
}
@Override
public
String
toString
()
{
return
"ReleaseOnConsumptionResultPartition "
+
partitionId
.
toString
()
+
" ["
+
partitionType
+
", "
+
subpartitions
.
length
+
" subpartitions, "
+
pendingReferences
+
" pending reference
s]"
;
+
numUnconsumedSubpartitions
+
" pending consumption
s]"
;
}
}
flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/ReleaseOnConsumptionResultPartitionTest.java
浏览文件 @
acd15d25
...
...
@@ -21,8 +21,8 @@ import org.apache.flink.util.TestLogger;
import
org.junit.Test
;
import
static
org
.
hamcrest
.
Matchers
.
is
;
import
static
org
.
junit
.
Assert
.
assertT
hat
;
import
static
org
.
junit
.
Assert
.
assertFalse
;
import
static
org
.
junit
.
Assert
.
assertT
rue
;
/**
* Tests for the {@link ReleaseOnConsumptionResultPartitionTest}.
...
...
@@ -41,9 +41,42 @@ public class ReleaseOnConsumptionResultPartitionTest extends TestLogger {
manager
.
registerResultPartition
(
partition
);
partition
.
onConsumedSubpartition
(
0
);
assert
That
(
partition
.
isReleased
(),
is
(
false
));
assert
False
(
partition
.
isReleased
(
));
partition
.
onConsumedSubpartition
(
1
);
assertThat
(
partition
.
isReleased
(),
is
(
true
));
assertTrue
(
partition
.
isReleased
());
}
@Test
public
void
testMultipleReleaseCallsAreIdempotent
()
{
final
ResultPartitionManager
manager
=
new
ResultPartitionManager
();
final
ResultPartition
partition
=
new
ResultPartitionBuilder
()
.
setNumberOfSubpartitions
(
2
)
.
isReleasedOnConsumption
(
true
)
.
setResultPartitionManager
(
manager
)
.
build
();
manager
.
registerResultPartition
(
partition
);
partition
.
onConsumedSubpartition
(
0
);
partition
.
onConsumedSubpartition
(
0
);
assertFalse
(
partition
.
isReleased
());
}
@Test
public
void
testReleaseAfterIdempotentCalls
()
{
final
ResultPartitionManager
manager
=
new
ResultPartitionManager
();
final
ResultPartition
partition
=
new
ResultPartitionBuilder
()
.
setNumberOfSubpartitions
(
2
)
.
isReleasedOnConsumption
(
true
)
.
setResultPartitionManager
(
manager
)
.
build
();
manager
.
registerResultPartition
(
partition
);
partition
.
onConsumedSubpartition
(
0
);
partition
.
onConsumedSubpartition
(
0
);
partition
.
onConsumedSubpartition
(
1
);
assertTrue
(
partition
.
isReleased
());
}
}
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录