Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
doujutun3207
flink
提交
a3a7350d
F
flink
项目概览
doujutun3207
/
flink
与 Fork 源项目一致
从无法访问的项目Fork
通知
24
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
F
flink
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
a3a7350d
编写于
2月 11, 2015
作者:
Z
zentol
提交者:
Stephan Ewen
5月 12, 2015
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
[FLINK-785] ChainedAllReduce
上级
b4152d75
变更
2
隐藏空白更改
内联
并排
Showing
2 changed file
with
114 addition
and
1 deletion
+114
-1
flink-runtime/src/main/java/org/apache/flink/runtime/operators/DriverStrategy.java
...va/org/apache/flink/runtime/operators/DriverStrategy.java
+2
-1
flink-runtime/src/main/java/org/apache/flink/runtime/operators/chaining/ChainedAllReduceDriver.java
...nk/runtime/operators/chaining/ChainedAllReduceDriver.java
+112
-0
未找到文件。
flink-runtime/src/main/java/org/apache/flink/runtime/operators/DriverStrategy.java
浏览文件 @
a3a7350d
...
...
@@ -21,6 +21,7 @@ package org.apache.flink.runtime.operators;
import
static
org
.
apache
.
flink
.
runtime
.
operators
.
DamBehavior
.
FULL_DAM
;
import
static
org
.
apache
.
flink
.
runtime
.
operators
.
DamBehavior
.
MATERIALIZING
;
import
static
org
.
apache
.
flink
.
runtime
.
operators
.
DamBehavior
.
PIPELINED
;
import
org.apache.flink.runtime.operators.chaining.ChainedAllReduceDriver
;
import
org.apache.flink.runtime.operators.chaining.ChainedCollectorMapDriver
;
import
org.apache.flink.runtime.operators.chaining.ChainedDriver
;
...
...
@@ -51,7 +52,7 @@ public enum DriverStrategy {
FLAT_MAP
(
FlatMapDriver
.
class
,
ChainedFlatMapDriver
.
class
,
PIPELINED
,
0
),
// group everything together into one group and apply the Reduce function
ALL_REDUCE
(
AllReduceDriver
.
class
,
null
,
PIPELINED
,
0
),
ALL_REDUCE
(
AllReduceDriver
.
class
,
ChainedAllReduceDriver
.
class
,
PIPELINED
,
0
),
// group everything together into one group and apply the GroupReduce function
ALL_GROUP_REDUCE
(
AllGroupReduceDriver
.
class
,
null
,
PIPELINED
,
0
),
// group everything together into one group and apply the GroupReduce's combine function
...
...
flink-runtime/src/main/java/org/apache/flink/runtime/operators/chaining/ChainedAllReduceDriver.java
0 → 100644
浏览文件 @
a3a7350d
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package
org.apache.flink.runtime.operators.chaining
;
import
org.apache.flink.api.common.functions.Function
;
import
org.apache.flink.api.common.functions.ReduceFunction
;
import
org.apache.flink.api.common.functions.util.FunctionUtils
;
import
org.apache.flink.api.common.typeutils.TypeSerializer
;
import
org.apache.flink.api.common.typeutils.TypeSerializerFactory
;
import
org.apache.flink.configuration.Configuration
;
import
org.apache.flink.runtime.jobgraph.tasks.AbstractInvokable
;
import
org.apache.flink.runtime.operators.RegularPactTask
;
import
org.slf4j.Logger
;
import
org.slf4j.LoggerFactory
;
public
class
ChainedAllReduceDriver
<
IT
>
extends
ChainedDriver
<
IT
,
IT
>
{
private
static
final
Logger
LOG
=
LoggerFactory
.
getLogger
(
ChainedAllReduceDriver
.
class
);
// --------------------------------------------------------------------------------------------
private
ReduceFunction
<
IT
>
reducer
;
private
TypeSerializer
<
IT
>
serializer
;
private
IT
base
;
// --------------------------------------------------------------------------------------------
@Override
public
void
setup
(
AbstractInvokable
parent
)
{
@SuppressWarnings
(
"unchecked"
)
final
ReduceFunction
<
IT
>
red
=
RegularPactTask
.
instantiateUserCode
(
this
.
config
,
userCodeClassLoader
,
ReduceFunction
.
class
);
this
.
reducer
=
red
;
FunctionUtils
.
setFunctionRuntimeContext
(
red
,
getUdfRuntimeContext
());
TypeSerializerFactory
<
IT
>
serializerFactory
=
this
.
config
.
getInputSerializer
(
0
,
userCodeClassLoader
);
this
.
serializer
=
serializerFactory
.
getSerializer
();
if
(
LOG
.
isDebugEnabled
())
{
LOG
.
debug
(
"ChainedAllReduceDriver object reuse: "
+
(
this
.
objectReuseEnabled
?
"ENABLED"
:
"DISABLED"
)
+
"."
);
}
}
@Override
public
void
openTask
()
throws
Exception
{
Configuration
stubConfig
=
this
.
config
.
getStubParameters
();
RegularPactTask
.
openUserCode
(
this
.
reducer
,
stubConfig
);
}
@Override
public
void
closeTask
()
throws
Exception
{
RegularPactTask
.
closeUserCode
(
this
.
reducer
);
}
@Override
public
void
cancelTask
()
{
try
{
FunctionUtils
.
closeFunction
(
this
.
reducer
);
}
catch
(
Throwable
t
)
{
}
}
// --------------------------------------------------------------------------------------------
@Override
public
Function
getStub
()
{
return
this
.
reducer
;
}
@Override
public
String
getTaskName
()
{
return
this
.
taskName
;
}
// --------------------------------------------------------------------------------------------
@Override
public
void
collect
(
IT
record
)
{
try
{
if
(
base
==
null
)
{
base
=
objectReuseEnabled
?
record
:
serializer
.
copy
(
record
);
}
else
{
base
=
objectReuseEnabled
?
reducer
.
reduce
(
base
,
record
)
:
serializer
.
copy
(
reducer
.
reduce
(
base
,
record
));
}
}
catch
(
Exception
e
)
{
throw
new
ExceptionInChainedStubException
(
taskName
,
e
);
}
}
@Override
public
void
close
()
{
try
{
if
(
base
!=
null
)
{
this
.
outputCollector
.
collect
(
base
);
base
=
null
;
}
}
catch
(
Exception
e
)
{
throw
new
ExceptionInChainedStubException
(
this
.
taskName
,
e
);
}
this
.
outputCollector
.
close
();
}
}
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录