Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
doujutun3207
flink
提交
4cc6bb1d
F
flink
项目概览
doujutun3207
/
flink
与 Fork 源项目一致
从无法访问的项目Fork
通知
24
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
F
flink
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
4cc6bb1d
编写于
12月 12, 2014
作者:
S
Stephan Ewen
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
[FLINK-1324] [runtime] Trailing data is cached before the local input strategies are closed.
上级
e47365bc
变更
6
隐藏空白更改
内联
并排
Showing
6 changed file
with
198 addition
and
2 deletion
+198
-2
flink-runtime/src/main/java/org/apache/flink/runtime/operators/RegularPactTask.java
...a/org/apache/flink/runtime/operators/RegularPactTask.java
+11
-2
flink-tests/src/test/java/org/apache/flink/test/iterative/nephele/danglingpagerank/CompensatableDotProductCoGroup.java
...hele/danglingpagerank/CompensatableDotProductCoGroup.java
+1
-0
flink-tests/src/test/java/org/apache/flink/test/iterative/nephele/danglingpagerank/CompensatableDotProductMatch.java
...ephele/danglingpagerank/CompensatableDotProductMatch.java
+1
-0
flink-tests/src/test/java/org/apache/flink/test/iterative/nephele/danglingpagerank/CompensatingMap.java
...t/iterative/nephele/danglingpagerank/CompensatingMap.java
+1
-0
flink-tests/src/test/java/org/apache/flink/test/misc/IterationIncompleteDynamicPathConsumptionITCase.java
...misc/IterationIncompleteDynamicPathConsumptionITCase.java
+92
-0
flink-tests/src/test/java/org/apache/flink/test/misc/IterationIncompleteStaticPathConsumptionITCase.java
.../misc/IterationIncompleteStaticPathConsumptionITCase.java
+92
-0
未找到文件。
flink-runtime/src/main/java/org/apache/flink/runtime/operators/RegularPactTask.java
浏览文件 @
4cc6bb1d
...
...
@@ -855,8 +855,8 @@ public class RegularPactTask<S extends Function, OT> extends AbstractInvokable i
for
(
int
i
=
0
;
i
<
numInputs
;
i
++)
{
final
int
memoryPages
;
final
boolean
async
=
this
.
config
.
isInputAsynchronouslyMaterialized
(
i
);
final
boolean
cached
=
this
.
config
.
isInputCached
(
i
);
final
boolean
async
=
this
.
config
.
isInputAsynchronouslyMaterialized
(
i
);
this
.
inputIsAsyncMaterialized
[
i
]
=
async
;
this
.
inputIsCached
[
i
]
=
cached
;
...
...
@@ -888,6 +888,16 @@ public class RegularPactTask<S extends Function, OT> extends AbstractInvokable i
}
protected
void
resetAllInputs
()
throws
Exception
{
// first we need to make sure that caches consume remaining data
// NOTE: we need to do this before closing the local strategies
for
(
int
i
=
0
;
i
<
this
.
inputs
.
length
;
i
++)
{
if
(
this
.
inputIsCached
[
i
]
&&
this
.
resettableInputs
[
i
]
!=
null
)
{
this
.
resettableInputs
[
i
].
consumeAndCacheRemainingData
();
}
}
// close all local-strategies. they will either get re-initialized, or we have
// read them now and their data is cached
for
(
int
i
=
0
;
i
<
this
.
localStrategies
.
length
;
i
++)
{
...
...
@@ -918,7 +928,6 @@ public class RegularPactTask<S extends Function, OT> extends AbstractInvokable i
if
(
this
.
tempBarriers
[
i
]
!=
null
)
{
this
.
inputs
[
i
]
=
this
.
tempBarriers
[
i
].
getIterator
();
}
else
if
(
this
.
resettableInputs
[
i
]
!=
null
)
{
this
.
resettableInputs
[
i
].
consumeAndCacheRemainingData
();
this
.
resettableInputs
[
i
].
reset
();
this
.
inputs
[
i
]
=
this
.
resettableInputs
[
i
];
}
else
{
...
...
flink-tests/src/test/java/org/apache/flink/test/iterative/nephele/danglingpagerank/CompensatableDotProductCoGroup.java
浏览文件 @
4cc6bb1d
...
...
@@ -29,6 +29,7 @@ import org.apache.flink.types.LongValue;
import
org.apache.flink.types.Record
;
import
org.apache.flink.util.Collector
;
@SuppressWarnings
(
"deprecation"
)
public
class
CompensatableDotProductCoGroup
extends
CoGroupFunction
{
private
static
final
long
serialVersionUID
=
1L
;
...
...
flink-tests/src/test/java/org/apache/flink/test/iterative/nephele/danglingpagerank/CompensatableDotProductMatch.java
浏览文件 @
4cc6bb1d
...
...
@@ -30,6 +30,7 @@ import org.apache.flink.types.LongValue;
import
org.apache.flink.types.Record
;
import
org.apache.flink.util.Collector
;
@SuppressWarnings
(
"deprecation"
)
public
class
CompensatableDotProductMatch
extends
JoinFunction
{
private
static
final
long
serialVersionUID
=
1L
;
...
...
flink-tests/src/test/java/org/apache/flink/test/iterative/nephele/danglingpagerank/CompensatingMap.java
浏览文件 @
4cc6bb1d
...
...
@@ -28,6 +28,7 @@ import org.apache.flink.types.DoubleValue;
import
org.apache.flink.types.Record
;
import
org.apache.flink.util.Collector
;
@SuppressWarnings
(
"deprecation"
)
public
class
CompensatingMap
extends
MapFunction
{
private
static
final
long
serialVersionUID
=
1L
;
...
...
flink-tests/src/test/java/org/apache/flink/test/misc/IterationIncompleteDynamicPathConsumptionITCase.java
0 → 100644
浏览文件 @
4cc6bb1d
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package
org.apache.flink.test.misc
;
import
org.apache.flink.api.common.functions.JoinFunction
;
import
org.apache.flink.api.common.operators.base.JoinOperatorBase.JoinHint
;
import
org.apache.flink.api.java.DataSet
;
import
org.apache.flink.api.java.ExecutionEnvironment
;
import
org.apache.flink.api.java.io.DiscardingOuputFormat
;
import
org.apache.flink.api.java.operators.IterativeDataSet
;
import
org.apache.flink.test.util.JavaProgramTestBase
;
@SuppressWarnings
(
"serial"
)
public
class
IterationIncompleteDynamicPathConsumptionITCase
extends
JavaProgramTestBase
{
@Override
protected
void
testProgram
()
throws
Exception
{
ExecutionEnvironment
env
=
ExecutionEnvironment
.
getExecutionEnvironment
();
// the test data is constructed such that the merge join zig zag
// has an early out, leaving elements on the dynamic path input unconsumed
DataSet
<
Path
>
edges
=
env
.
fromElements
(
new
Path
(
1
,
2
),
new
Path
(
1
,
4
),
new
Path
(
3
,
6
),
new
Path
(
3
,
8
),
new
Path
(
1
,
10
),
new
Path
(
1
,
12
),
new
Path
(
3
,
14
),
new
Path
(
3
,
16
),
new
Path
(
1
,
18
),
new
Path
(
1
,
20
)
);
IterativeDataSet
<
Path
>
currentPaths
=
edges
.
iterate
(
10
);
DataSet
<
Path
>
newPaths
=
currentPaths
.
join
(
edges
,
JoinHint
.
REPARTITION_SORT_MERGE
).
where
(
"to"
).
equalTo
(
"from"
)
.
with
(
new
PathConnector
())
.
union
(
currentPaths
).
distinct
(
"from"
,
"to"
);
DataSet
<
Path
>
result
=
currentPaths
.
closeWith
(
newPaths
);
result
.
output
(
new
DiscardingOuputFormat
<
Path
>());
env
.
execute
();
}
private
static
class
PathConnector
implements
JoinFunction
<
Path
,
Path
,
Path
>
{
@Override
public
Path
join
(
Path
path
,
Path
edge
)
{
return
new
Path
(
path
.
from
,
edge
.
to
);
}
}
// --------------------------------------------------------------------------------------------
public
static
class
Path
{
public
long
from
;
public
long
to
;
public
Path
()
{}
public
Path
(
long
from
,
long
to
)
{
this
.
from
=
from
;
this
.
to
=
to
;
}
@Override
public
String
toString
()
{
return
"("
+
from
+
","
+
to
+
")"
;
}
}
}
flink-tests/src/test/java/org/apache/flink/test/misc/IterationIncompleteStaticPathConsumptionITCase.java
0 → 100644
浏览文件 @
4cc6bb1d
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package
org.apache.flink.test.misc
;
import
org.apache.flink.api.common.functions.JoinFunction
;
import
org.apache.flink.api.common.operators.base.JoinOperatorBase.JoinHint
;
import
org.apache.flink.api.java.DataSet
;
import
org.apache.flink.api.java.ExecutionEnvironment
;
import
org.apache.flink.api.java.io.DiscardingOuputFormat
;
import
org.apache.flink.api.java.operators.IterativeDataSet
;
import
org.apache.flink.test.util.JavaProgramTestBase
;
@SuppressWarnings
(
"serial"
)
public
class
IterationIncompleteStaticPathConsumptionITCase
extends
JavaProgramTestBase
{
@Override
protected
void
testProgram
()
throws
Exception
{
ExecutionEnvironment
env
=
ExecutionEnvironment
.
getExecutionEnvironment
();
// the test data is constructed such that the merge join zig zag
// has an early out, leaving elements on the static path input unconsumed
DataSet
<
Path
>
edges
=
env
.
fromElements
(
new
Path
(
2
,
1
),
new
Path
(
4
,
1
),
new
Path
(
6
,
3
),
new
Path
(
8
,
3
),
new
Path
(
10
,
1
),
new
Path
(
12
,
1
),
new
Path
(
14
,
3
),
new
Path
(
16
,
3
),
new
Path
(
18
,
1
),
new
Path
(
20
,
1
)
);
IterativeDataSet
<
Path
>
currentPaths
=
edges
.
iterate
(
10
);
DataSet
<
Path
>
newPaths
=
currentPaths
.
join
(
edges
,
JoinHint
.
REPARTITION_SORT_MERGE
).
where
(
"to"
).
equalTo
(
"from"
)
.
with
(
new
PathConnector
())
.
union
(
currentPaths
).
distinct
(
"from"
,
"to"
);
DataSet
<
Path
>
result
=
currentPaths
.
closeWith
(
newPaths
);
result
.
output
(
new
DiscardingOuputFormat
<
Path
>());
env
.
execute
();
}
private
static
class
PathConnector
implements
JoinFunction
<
Path
,
Path
,
Path
>
{
@Override
public
Path
join
(
Path
path
,
Path
edge
)
{
return
new
Path
(
path
.
from
,
edge
.
to
);
}
}
// --------------------------------------------------------------------------------------------
public
static
class
Path
{
public
long
from
;
public
long
to
;
public
Path
()
{}
public
Path
(
long
from
,
long
to
)
{
this
.
from
=
from
;
this
.
to
=
to
;
}
@Override
public
String
toString
()
{
return
"("
+
from
+
","
+
to
+
")"
;
}
}
}
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录