Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
doujutun3207
flink
提交
b2f8e307
F
flink
项目概览
doujutun3207
/
flink
与 Fork 源项目一致
从无法访问的项目Fork
通知
24
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
F
flink
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
b2f8e307
编写于
8月 19, 2015
作者:
U
Ufuk Celebi
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
[FLINK-2089] [runtime] Fix illegal state in RecordWriter after partition write failure
- Address PR comments This closes #1050.
上级
6f07c5f3
变更
2
隐藏空白更改
内联
并排
Showing
2 changed file
with
162 addition
and
114 deletion
+162
-114
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/api/writer/RecordWriter.java
...che/flink/runtime/io/network/api/writer/RecordWriter.java
+39
-20
flink-runtime/src/test/java/org/apache/flink/runtime/io/network/api/writer/RecordWriterTest.java
...flink/runtime/io/network/api/writer/RecordWriterTest.java
+123
-94
未找到文件。
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/api/writer/RecordWriter.java
浏览文件 @
b2f8e307
...
...
@@ -86,8 +86,7 @@ public class RecordWriter<T extends IOReadableWritable> {
Buffer
buffer
=
serializer
.
getCurrentBuffer
();
if
(
buffer
!=
null
)
{
writer
.
writeBuffer
(
buffer
,
targetChannel
);
serializer
.
clearCurrentBuffer
();
writeBuffer
(
buffer
,
targetChannel
,
serializer
);
}
buffer
=
writer
.
getBufferProvider
().
requestBufferBlocking
();
...
...
@@ -112,8 +111,7 @@ public class RecordWriter<T extends IOReadableWritable> {
Buffer
buffer
=
serializer
.
getCurrentBuffer
();
if
(
buffer
!=
null
)
{
writer
.
writeBuffer
(
buffer
,
targetChannel
);
serializer
.
clearCurrentBuffer
();
writeBuffer
(
buffer
,
targetChannel
,
serializer
);
}
buffer
=
writer
.
getBufferProvider
().
requestBufferBlocking
();
...
...
@@ -135,8 +133,7 @@ public class RecordWriter<T extends IOReadableWritable> {
throw
new
IllegalStateException
(
"Serializer has data but no buffer."
);
}
writer
.
writeBuffer
(
buffer
,
targetChannel
);
serializer
.
clearCurrentBuffer
();
writeBuffer
(
buffer
,
targetChannel
,
serializer
);
writer
.
writeEvent
(
event
,
targetChannel
);
...
...
@@ -157,8 +154,7 @@ public class RecordWriter<T extends IOReadableWritable> {
synchronized
(
serializer
)
{
Buffer
buffer
=
serializer
.
getCurrentBuffer
();
if
(
buffer
!=
null
)
{
writer
.
writeBuffer
(
buffer
,
targetChannel
);
serializer
.
clearCurrentBuffer
();
writeBuffer
(
buffer
,
targetChannel
,
serializer
);
buffer
=
writer
.
getBufferProvider
().
requestBufferBlocking
();
serializer
.
setNextBuffer
(
buffer
);
...
...
@@ -174,26 +170,31 @@ public class RecordWriter<T extends IOReadableWritable> {
RecordSerializer
<
T
>
serializer
=
serializers
[
targetChannel
];
synchronized
(
serializer
)
{
Buffer
buffer
=
serializer
.
getCurrentBuffer
();
try
{
Buffer
buffer
=
serializer
.
getCurrentBuffer
();
if
(
buffer
!=
null
)
{
// Only clear the serializer after the buffer was written out.
writer
.
writeBuffer
(
buffer
,
targetChannel
);
if
(
buffer
!=
null
)
{
writeBuffer
(
buffer
,
targetChannel
,
serializer
);
}
}
finally
{
serializer
.
clear
();
}
serializer
.
clear
();
}
}
}
public
void
clearBuffers
()
{
for
(
RecordSerializer
<?>
s
:
serializers
)
{
synchronized
(
s
)
{
Buffer
b
=
s
.
getCurrentBuffer
();
s
.
clea
r
();
for
(
RecordSerializer
<?>
s
erializer
:
serializers
)
{
synchronized
(
s
erializer
)
{
try
{
Buffer
buffer
=
serializer
.
getCurrentBuffe
r
();
if
(
b
!=
null
)
{
b
.
recycle
();
if
(
buffer
!=
null
)
{
buffer
.
recycle
();
}
}
finally
{
serializer
.
clear
();
}
}
}
...
...
@@ -208,4 +209,22 @@ public class RecordWriter<T extends IOReadableWritable> {
}
}
/**
* Writes the buffer to the {@link ResultPartitionWriter}.
*
* <p> The buffer is cleared from the serializer state after a call to this method.
*/
private
void
writeBuffer
(
Buffer
buffer
,
int
targetChannel
,
RecordSerializer
<
T
>
serializer
)
throws
IOException
{
try
{
writer
.
writeBuffer
(
buffer
,
targetChannel
);
}
finally
{
serializer
.
clearCurrentBuffer
();
}
}
}
flink-runtime/src/test/java/org/apache/flink/runtime/io/network/api/writer/RecordWriterTest.java
浏览文件 @
b2f8e307
...
...
@@ -19,9 +19,13 @@
package
org.apache.flink.runtime.io.network.api.writer
;
import
org.apache.flink.runtime.io.network.buffer.Buffer
;
import
org.apache.flink.runtime.io.network.buffer.BufferPool
;
import
org.apache.flink.runtime.io.network.buffer.BufferProvider
;
import
org.apache.flink.runtime.io.network.buffer.NetworkBufferPool
;
import
org.apache.flink.runtime.io.network.util.TestBufferFactory
;
import
org.apache.flink.runtime.io.network.util.TestTaskEvent
;
import
org.apache.flink.types.IntValue
;
import
org.junit.Assert
;
import
org.junit.Test
;
import
org.junit.runner.RunWith
;
import
org.mockito.invocation.InvocationOnMock
;
...
...
@@ -37,6 +41,7 @@ import java.util.concurrent.Executors;
import
java.util.concurrent.Future
;
import
static
com
.
google
.
common
.
base
.
Preconditions
.
checkNotNull
;
import
static
org
.
junit
.
Assert
.
assertEquals
;
import
static
org
.
junit
.
Assert
.
assertTrue
;
import
static
org
.
junit
.
Assert
.
fail
;
import
static
org
.
mockito
.
Matchers
.
any
;
...
...
@@ -59,100 +64,6 @@ public class RecordWriterTest {
// Resource release tests
// ---------------------------------------------------------------------------------------------
@Test
public
void
testClearBuffersAfterEmit
()
throws
Exception
{
final
Buffer
buffer
=
TestBufferFactory
.
createBuffer
(
32
);
BufferProvider
bufferProvider
=
createBufferProvider
(
buffer
);
ResultPartitionWriter
partitionWriter
=
createResultPartitionWriter
(
bufferProvider
);
RecordWriter
<
IntValue
>
recordWriter
=
new
RecordWriter
<
IntValue
>(
partitionWriter
);
// Emit single record, the buffer will not be written out.
// Therefore, it needs to be cleared explicitly.
recordWriter
.
emit
(
new
IntValue
(
0
));
// Verify that a buffer is requested, but not written out.
verify
(
bufferProvider
,
times
(
1
)).
requestBufferBlocking
();
verify
(
partitionWriter
,
never
()).
writeBuffer
(
any
(
Buffer
.
class
),
anyInt
());
recordWriter
.
clearBuffers
();
assertTrue
(
"Buffer not recycled."
,
buffer
.
isRecycled
());
}
@Test
public
void
testClearBuffersAfterExceptionInFlushWritePartition
()
throws
Exception
{
// Size of buffer ensures that a single record will fill the buffer.
final
Buffer
buffer
=
TestBufferFactory
.
createBuffer
(
4
);
BufferProvider
bufferProvider
=
createBufferProvider
(
buffer
);
ResultPartitionWriter
partitionWriter
=
createResultPartitionWriter
(
bufferProvider
);
doThrow
(
new
IOException
(
"Expected test exception"
))
.
when
(
partitionWriter
).
writeBuffer
(
eq
(
buffer
),
eq
(
0
));
RecordWriter
<
IntValue
>
recordWriter
=
new
RecordWriter
<
IntValue
>(
partitionWriter
);
try
{
// Emit single record, the buffer will not be written out,
// because of the Exception. Therefore, it needs to be cleared
// explicitly.
recordWriter
.
emit
(
new
IntValue
(
0
));
fail
(
"Did not throw expected Exception. This means that the record "
+
"writer did not request a buffer as expected."
);
}
catch
(
IOException
expected
)
{
}
// Verify that a buffer is requested, but not written out due to the Exception.
verify
(
bufferProvider
,
times
(
1
)).
requestBufferBlocking
();
verify
(
partitionWriter
,
times
(
1
)).
writeBuffer
(
any
(
Buffer
.
class
),
anyInt
());
recordWriter
.
clearBuffers
();
assertTrue
(
"Buffer not recycled."
,
buffer
.
isRecycled
());
}
@Test
public
void
testClearBuffersAfterExceptionInEmitWritePartition
()
throws
Exception
{
// Size of buffer ensures that a single record will NOT fill the buffer.
final
Buffer
buffer
=
TestBufferFactory
.
createBuffer
(
32
);
BufferProvider
bufferProvider
=
createBufferProvider
(
buffer
);
ResultPartitionWriter
partitionWriter
=
createResultPartitionWriter
(
bufferProvider
);
doThrow
(
new
IOException
(
"Expected test exception"
))
.
when
(
partitionWriter
).
writeBuffer
(
eq
(
buffer
),
eq
(
0
));
RecordWriter
<
IntValue
>
recordWriter
=
new
RecordWriter
<
IntValue
>(
partitionWriter
);
try
{
recordWriter
.
emit
(
new
IntValue
(
0
));
// Verify that a buffer is requested, but not written out.
verify
(
bufferProvider
,
times
(
1
)).
requestBufferBlocking
();
verify
(
partitionWriter
,
never
()).
writeBuffer
(
any
(
Buffer
.
class
),
anyInt
());
// Now flush the record.
recordWriter
.
flush
();
fail
(
"Did not throw expected Exception. This means that the record "
+
"writer did not request a buffer as expected."
);
}
catch
(
IOException
expected
)
{
}
// Flushing the buffer tried to write out the buffer.
verify
(
partitionWriter
,
times
(
1
)).
writeBuffer
(
any
(
Buffer
.
class
),
anyInt
());
recordWriter
.
clearBuffers
();
assertTrue
(
"Buffer not recycled."
,
buffer
.
isRecycled
());
}
/**
* Tests a fix for FLINK-2089.
*
...
...
@@ -241,6 +152,124 @@ public class RecordWriterTest {
}
}
@Test
public
void
testClearBuffersAfterExceptionInPartitionWriter
()
throws
Exception
{
NetworkBufferPool
buffers
=
null
;
BufferPool
bufferPool
=
null
;
try
{
buffers
=
new
NetworkBufferPool
(
1
,
1024
);
bufferPool
=
spy
(
buffers
.
createBufferPool
(
1
,
true
));
ResultPartitionWriter
partitionWriter
=
mock
(
ResultPartitionWriter
.
class
);
when
(
partitionWriter
.
getBufferProvider
()).
thenReturn
(
checkNotNull
(
bufferPool
));
when
(
partitionWriter
.
getNumberOfOutputChannels
()).
thenReturn
(
1
);
// Recycle buffer and throw Exception
doAnswer
(
new
Answer
<
Void
>()
{
@Override
public
Void
answer
(
InvocationOnMock
invocation
)
throws
Throwable
{
Buffer
buffer
=
(
Buffer
)
invocation
.
getArguments
()[
0
];
buffer
.
recycle
();
throw
new
RuntimeException
(
"Expected test Exception"
);
}
}).
when
(
partitionWriter
).
writeBuffer
(
any
(
Buffer
.
class
),
anyInt
());
RecordWriter
<
IntValue
>
recordWriter
=
new
RecordWriter
<>(
partitionWriter
);
try
{
// Verify that emit correctly clears the buffer. The infinite loop looks
// dangerous indeed, but the buffer will only be flushed after its full. Adding a
// manual flush here doesn't test this case (see next).
for
(;;)
{
recordWriter
.
emit
(
new
IntValue
(
0
));
}
}
catch
(
Exception
e
)
{
// Verify that the buffer is not part of the record writer state after a failure
// to flush it out. If the buffer is still part of the record writer state, this
// will fail, because the buffer has already been recycled. NOTE: The mock
// partition writer needs to recycle the buffer to correctly test this.
recordWriter
.
clearBuffers
();
}
// Verify expected methods have been called
verify
(
partitionWriter
,
times
(
1
)).
writeBuffer
(
any
(
Buffer
.
class
),
anyInt
());
verify
(
bufferPool
,
times
(
1
)).
requestBufferBlocking
();
try
{
// Verify that manual flushing correctly clears the buffer.
recordWriter
.
emit
(
new
IntValue
(
0
));
recordWriter
.
flush
();
Assert
.
fail
(
"Did not throw expected test Exception"
);
}
catch
(
Exception
e
)
{
recordWriter
.
clearBuffers
();
}
// Verify expected methods have been called
verify
(
partitionWriter
,
times
(
2
)).
writeBuffer
(
any
(
Buffer
.
class
),
anyInt
());
verify
(
bufferPool
,
times
(
2
)).
requestBufferBlocking
();
try
{
// Verify that broadcast emit correctly clears the buffer.
for
(;;)
{
recordWriter
.
broadcastEmit
(
new
IntValue
(
0
));
}
}
catch
(
Exception
e
)
{
recordWriter
.
clearBuffers
();
}
// Verify expected methods have been called
verify
(
partitionWriter
,
times
(
3
)).
writeBuffer
(
any
(
Buffer
.
class
),
anyInt
());
verify
(
bufferPool
,
times
(
3
)).
requestBufferBlocking
();
try
{
// Verify that end of super step correctly clears the buffer.
recordWriter
.
emit
(
new
IntValue
(
0
));
recordWriter
.
sendEndOfSuperstep
();
Assert
.
fail
(
"Did not throw expected test Exception"
);
}
catch
(
Exception
e
)
{
recordWriter
.
clearBuffers
();
}
// Verify expected methods have been called
verify
(
partitionWriter
,
times
(
4
)).
writeBuffer
(
any
(
Buffer
.
class
),
anyInt
());
verify
(
bufferPool
,
times
(
4
)).
requestBufferBlocking
();
try
{
// Verify that broadcasting and event correctly clears the buffer.
recordWriter
.
emit
(
new
IntValue
(
0
));
recordWriter
.
broadcastEvent
(
new
TestTaskEvent
());
Assert
.
fail
(
"Did not throw expected test Exception"
);
}
catch
(
Exception
e
)
{
recordWriter
.
clearBuffers
();
}
// Verify expected methods have been called
verify
(
partitionWriter
,
times
(
5
)).
writeBuffer
(
any
(
Buffer
.
class
),
anyInt
());
verify
(
bufferPool
,
times
(
5
)).
requestBufferBlocking
();
}
finally
{
if
(
bufferPool
!=
null
)
{
assertEquals
(
1
,
bufferPool
.
getNumberOfAvailableMemorySegments
());
bufferPool
.
lazyDestroy
();
}
if
(
buffers
!=
null
)
{
assertEquals
(
1
,
buffers
.
getNumberOfAvailableMemorySegments
());
buffers
.
destroy
();
}
}
}
@Test
public
void
testSerializerClearedAfterClearBuffers
()
throws
Exception
{
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录