Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
doujutun3207
flink
提交
3002258f
F
flink
项目概览
doujutun3207
/
flink
与 Fork 源项目一致
从无法访问的项目Fork
通知
24
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
F
flink
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
3002258f
编写于
7月 12, 2014
作者:
S
Stephan Ewen
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
[FLINK-1018] Fix cross pipelining/daming info to resolve cross-related streaming deadlocks.
上级
ec0b00d6
变更
2
隐藏空白更改
内联
并排
Showing
2 changed file
with
106 addition
and
5 deletion
+106
-5
flink-compiler/src/test/java/org/apache/flink/compiler/PipelineBreakerTest.java
...t/java/org/apache/flink/compiler/PipelineBreakerTest.java
+102
-1
flink-runtime/src/main/java/org/apache/flink/runtime/operators/DriverStrategy.java
...va/org/apache/flink/runtime/operators/DriverStrategy.java
+4
-4
未找到文件。
flink-compiler/src/test/java/org/apache/flink/compiler/PipelineBreakerTest.java
浏览文件 @
3002258f
...
@@ -23,12 +23,13 @@ import org.apache.flink.api.java.DataSet;
...
@@ -23,12 +23,13 @@ import org.apache.flink.api.java.DataSet;
import
org.apache.flink.api.java.ExecutionEnvironment
;
import
org.apache.flink.api.java.ExecutionEnvironment
;
import
org.apache.flink.api.java.IterativeDataSet
;
import
org.apache.flink.api.java.IterativeDataSet
;
import
org.apache.flink.compiler.plan.BulkIterationPlanNode
;
import
org.apache.flink.compiler.plan.BulkIterationPlanNode
;
import
org.apache.flink.compiler.plan.DualInputPlanNode
;
import
org.apache.flink.compiler.plan.OptimizedPlan
;
import
org.apache.flink.compiler.plan.OptimizedPlan
;
import
org.apache.flink.compiler.plan.SingleInputPlanNode
;
import
org.apache.flink.compiler.plan.SingleInputPlanNode
;
import
org.apache.flink.compiler.plan.SinkPlanNode
;
import
org.apache.flink.compiler.plan.SinkPlanNode
;
import
org.apache.flink.compiler.plandump.PlanJSONDumpGenerator
;
import
org.apache.flink.compiler.testfunctions.IdentityMapper
;
import
org.apache.flink.compiler.testfunctions.IdentityMapper
;
import
org.apache.flink.compiler.testfunctions.SelectOneReducer
;
import
org.apache.flink.compiler.testfunctions.SelectOneReducer
;
import
org.apache.flink.configuration.Configuration
;
@SuppressWarnings
(
"serial"
)
@SuppressWarnings
(
"serial"
)
public
class
PipelineBreakerTest
extends
CompilerTestBase
{
public
class
PipelineBreakerTest
extends
CompilerTestBase
{
...
@@ -134,4 +135,104 @@ public class PipelineBreakerTest extends CompilerTestBase {
...
@@ -134,4 +135,104 @@ public class PipelineBreakerTest extends CompilerTestBase {
fail
(
e
.
getMessage
());
fail
(
e
.
getMessage
());
}
}
}
}
@Test
public
void
testPilelineBreakerWithCross
()
{
try
{
{
ExecutionEnvironment
env
=
ExecutionEnvironment
.
getExecutionEnvironment
();
env
.
setDegreeOfParallelism
(
64
);
DataSet
<
Long
>
initialSource
=
env
.
generateSequence
(
1
,
10
);
Configuration
conf
=
new
Configuration
();
conf
.
setString
(
PactCompiler
.
HINT_LOCAL_STRATEGY
,
PactCompiler
.
HINT_LOCAL_STRATEGY_NESTEDLOOP_BLOCKED_OUTER_FIRST
);
initialSource
.
map
(
new
IdentityMapper
<
Long
>())
.
cross
(
initialSource
).
withParameters
(
conf
)
.
print
();
Plan
p
=
env
.
createProgramPlan
();
OptimizedPlan
op
=
compileNoStats
(
p
);
SinkPlanNode
sink
=
op
.
getDataSinks
().
iterator
().
next
();
DualInputPlanNode
mapper
=
(
DualInputPlanNode
)
sink
.
getInput
().
getSource
();
assertTrue
(
mapper
.
getInput1
().
getTempMode
().
breaksPipeline
());
}
{
ExecutionEnvironment
env
=
ExecutionEnvironment
.
getExecutionEnvironment
();
env
.
setDegreeOfParallelism
(
64
);
DataSet
<
Long
>
initialSource
=
env
.
generateSequence
(
1
,
10
);
Configuration
conf
=
new
Configuration
();
conf
.
setString
(
PactCompiler
.
HINT_LOCAL_STRATEGY
,
PactCompiler
.
HINT_LOCAL_STRATEGY_NESTEDLOOP_BLOCKED_OUTER_SECOND
);
initialSource
.
map
(
new
IdentityMapper
<
Long
>())
.
cross
(
initialSource
).
withParameters
(
conf
)
.
print
();
Plan
p
=
env
.
createProgramPlan
();
OptimizedPlan
op
=
compileNoStats
(
p
);
SinkPlanNode
sink
=
op
.
getDataSinks
().
iterator
().
next
();
DualInputPlanNode
mapper
=
(
DualInputPlanNode
)
sink
.
getInput
().
getSource
();
assertTrue
(
mapper
.
getInput2
().
getTempMode
().
breaksPipeline
());
}
{
ExecutionEnvironment
env
=
ExecutionEnvironment
.
getExecutionEnvironment
();
env
.
setDegreeOfParallelism
(
64
);
DataSet
<
Long
>
initialSource
=
env
.
generateSequence
(
1
,
10
);
Configuration
conf
=
new
Configuration
();
conf
.
setString
(
PactCompiler
.
HINT_LOCAL_STRATEGY
,
PactCompiler
.
HINT_LOCAL_STRATEGY_NESTEDLOOP_STREAMED_OUTER_FIRST
);
initialSource
.
map
(
new
IdentityMapper
<
Long
>())
.
cross
(
initialSource
).
withParameters
(
conf
)
.
print
();
Plan
p
=
env
.
createProgramPlan
();
OptimizedPlan
op
=
compileNoStats
(
p
);
SinkPlanNode
sink
=
op
.
getDataSinks
().
iterator
().
next
();
DualInputPlanNode
mapper
=
(
DualInputPlanNode
)
sink
.
getInput
().
getSource
();
assertTrue
(
mapper
.
getInput1
().
getTempMode
().
breaksPipeline
());
}
{
ExecutionEnvironment
env
=
ExecutionEnvironment
.
getExecutionEnvironment
();
env
.
setDegreeOfParallelism
(
64
);
DataSet
<
Long
>
initialSource
=
env
.
generateSequence
(
1
,
10
);
Configuration
conf
=
new
Configuration
();
conf
.
setString
(
PactCompiler
.
HINT_LOCAL_STRATEGY
,
PactCompiler
.
HINT_LOCAL_STRATEGY_NESTEDLOOP_STREAMED_OUTER_SECOND
);
initialSource
.
map
(
new
IdentityMapper
<
Long
>())
.
cross
(
initialSource
).
withParameters
(
conf
)
.
print
();
Plan
p
=
env
.
createProgramPlan
();
OptimizedPlan
op
=
compileNoStats
(
p
);
SinkPlanNode
sink
=
op
.
getDataSinks
().
iterator
().
next
();
DualInputPlanNode
mapper
=
(
DualInputPlanNode
)
sink
.
getInput
().
getSource
();
assertTrue
(
mapper
.
getInput2
().
getTempMode
().
breaksPipeline
());
}
}
catch
(
Exception
e
)
{
e
.
printStackTrace
();
fail
(
e
.
getMessage
());
}
}
}
}
flink-runtime/src/main/java/org/apache/flink/runtime/operators/DriverStrategy.java
浏览文件 @
3002258f
...
@@ -80,13 +80,13 @@ public enum DriverStrategy {
...
@@ -80,13 +80,13 @@ public enum DriverStrategy {
HYBRIDHASH_BUILD_SECOND_CACHED
(
BuildSecondCachedMatchDriver
.
class
,
null
,
MATERIALIZING
,
FULL_DAM
,
true
),
HYBRIDHASH_BUILD_SECOND_CACHED
(
BuildSecondCachedMatchDriver
.
class
,
null
,
MATERIALIZING
,
FULL_DAM
,
true
),
// the second input is inner loop, the first input is outer loop and block-wise processed
// the second input is inner loop, the first input is outer loop and block-wise processed
NESTEDLOOP_BLOCKED_OUTER_FIRST
(
CrossDriver
.
class
,
null
,
MATERIALIZING
,
MATERIALIZING
,
false
),
NESTEDLOOP_BLOCKED_OUTER_FIRST
(
CrossDriver
.
class
,
null
,
MATERIALIZING
,
FULL_DAM
,
false
),
// the first input is inner loop, the second input is outer loop and block-wise processed
// the first input is inner loop, the second input is outer loop and block-wise processed
NESTEDLOOP_BLOCKED_OUTER_SECOND
(
CrossDriver
.
class
,
null
,
MATERIALIZING
,
MATERIALIZING
,
false
),
NESTEDLOOP_BLOCKED_OUTER_SECOND
(
CrossDriver
.
class
,
null
,
FULL_DAM
,
MATERIALIZING
,
false
),
// the second input is inner loop, the first input is outer loop and stream-processed
// the second input is inner loop, the first input is outer loop and stream-processed
NESTEDLOOP_STREAMED_OUTER_FIRST
(
CrossDriver
.
class
,
null
,
PIPELINED
,
MATERIALIZING
,
false
),
NESTEDLOOP_STREAMED_OUTER_FIRST
(
CrossDriver
.
class
,
null
,
PIPELINED
,
FULL_DAM
,
false
),
// the first input is inner loop, the second input is outer loop and stream-processed
// the first input is inner loop, the second input is outer loop and stream-processed
NESTEDLOOP_STREAMED_OUTER_SECOND
(
CrossDriver
.
class
,
null
,
MATERIALIZING
,
PIPELINED
,
false
),
NESTEDLOOP_STREAMED_OUTER_SECOND
(
CrossDriver
.
class
,
null
,
FULL_DAM
,
PIPELINED
,
false
),
// union utility op. unions happen implicitly on the network layer (in the readers) when bundeling streams
// union utility op. unions happen implicitly on the network layer (in the readers) when bundeling streams
UNION
(
null
,
null
,
FULL_DAM
,
FULL_DAM
,
false
);
UNION
(
null
,
null
,
FULL_DAM
,
FULL_DAM
,
false
);
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录