Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
doujutun3207
flink
提交
daad5778
F
flink
项目概览
doujutun3207
/
flink
与 Fork 源项目一致
从无法访问的项目Fork
通知
24
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
F
flink
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
daad5778
编写于
12月 13, 2015
作者:
S
Stephan Ewen
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
[hotfix] Add comments to BroadcastVariableInitializer and SplittableIterator
上级
f414e55e
变更
4
隐藏空白更改
内联
并排
Showing
4 changed file
with
90 addition
and
5 deletion
+90
-5
flink-core/src/main/java/org/apache/flink/api/common/functions/BroadcastVariableInitializer.java
...nk/api/common/functions/BroadcastVariableInitializer.java
+60
-1
flink-core/src/main/java/org/apache/flink/core/memory/MemorySegmentSource.java
...ava/org/apache/flink/core/memory/MemorySegmentSource.java
+0
-2
flink-core/src/main/java/org/apache/flink/util/NumberSequenceIterator.java
...in/java/org/apache/flink/util/NumberSequenceIterator.java
+4
-1
flink-core/src/main/java/org/apache/flink/util/SplittableIterator.java
...c/main/java/org/apache/flink/util/SplittableIterator.java
+26
-1
未找到文件。
flink-core/src/main/java/org/apache/flink/api/common/functions/BroadcastVariableInitializer.java
浏览文件 @
daad5778
...
...
@@ -18,7 +18,66 @@
package
org.apache.flink.api.common.functions
;
/**
* A broadcast variable initializer can be used to transform a broadcast variable
* into another format during initialization. The transformed variable is shared
* among the parallel instances of a function inside one TaskManager, the
* same way as the plain broadcast variables (lists) are shared.
*
* <p>The broadcast variable initializer will in many cases only be executed by one
* parallel function instance per TaskManager, when acquiring the broadcast variable
* for the first time inside that particular TaskManager. It is possible that a
* broadcast variable is read and initialized multiple times, if the tasks that use
* the variables are not overlapping in their execution time; in such cases, it can
* happen that one function instance released the broadcast variable, and another
* function instance materializes it again.
*
* <p>This is an example of how to use a broadcast variable initializer, transforming
* the shared list of values into a shared map.
*
* <pre>{@code
* public class MyFunction extends RichMapFunction<Long, String> {
*
* private Map<Long, String> map;
*
* public void open(Configuration cfg) throws Exception {
* getRuntimeContext().getBroadcastVariableWithInitializer("mapvar",
* new BroadcastVariableInitializer<Tuple2<Long, String>, Map<Long, String>>() {
*
* public Map<Long, String> initializeBroadcastVariable(Iterable<Tuple2<Long, String>> data) {
* Map<Long, String> map = new HashMap<>();
*
* for (Tuple2<Long, String> t : data) {
* map.put(t.f0, t.f1);
* }
*
* return map;
* }
* });
* }
*
* public String map(Long value) {
* // replace the long by the String, based on the map
* return map.get(value);
* }
* }
*
* }</pre>
*
* @param <T> The type of the elements in the list of the original untransformed broadcast variable.
* @param <O> The type of the transformed broadcast variable.
*/
public
interface
BroadcastVariableInitializer
<
T
,
O
>
{
/**
* The method that reads the data elements from the broadcast variable and
* creates the transformed data structure.
*
* <p>The iterable with the data elements can be traversed only once, i.e.,
* only the first call to {@code iterator()} will succeed.
*
* @param data The sequence of elements in the broadcast variable.
* @return The transformed broadcast variable.
*/
O
initializeBroadcastVariable
(
Iterable
<
T
>
data
);
}
flink-core/src/main/java/org/apache/flink/core/memory/MemorySegmentSource.java
浏览文件 @
daad5778
...
...
@@ -16,10 +16,8 @@
* limitations under the License.
*/
package
org.apache.flink.core.memory
;
/**
* Interface describing entities that can provide memory segments.
*/
...
...
flink-core/src/main/java/org/apache/flink/util/NumberSequenceIterator.java
浏览文件 @
daad5778
...
...
@@ -29,13 +29,16 @@ public class NumberSequenceIterator extends SplittableIterator<Long> {
private
static
final
long
serialVersionUID
=
1L
;
/** The last number returned by the iterator */
private
final
long
to
;
/** The next number to be returned */
private
long
current
;
/**
* Internal constructor to allow for empty iterators.
* Creates a new splittable iterator, returning the range [from, to].
* Both boundaries of the interval are inclusive.
*
* @param from The first number returned by the iterator.
* @param to The last number returned by the iterator.
...
...
flink-core/src/main/java/org/apache/flink/util/SplittableIterator.java
浏览文件 @
daad5778
...
...
@@ -21,13 +21,33 @@ package org.apache.flink.util;
import
java.io.Serializable
;
import
java.util.Iterator
;
/**
* Abstract base class for iterators that can split themselves into multiple disjoint
* iterators. The union of these iterators returns the original iterator values.
*
* @param <T> The type of elements returned by the iterator.
*/
public
abstract
class
SplittableIterator
<
T
>
implements
Iterator
<
T
>,
Serializable
{
private
static
final
long
serialVersionUID
=
200377674313072307L
;
/**
* Splits this iterator into a number disjoint iterators.
* The union of these iterators returns the original iterator values.
*
* @param numPartitions The number of iterators to split into.
* @return An array with the split iterators.
*/
public
abstract
Iterator
<
T
>[]
split
(
int
numPartitions
);
/**
* Splits this iterator into <i>n</i> partitions and returns the <i>i-th</i> partition
* out of those.
*
* @param num The partition to return (<i>i</i>).
* @param numPartitions The number of partitions to split into (<i>n</i>).
* @return The iterator for the partition.
*/
public
Iterator
<
T
>
getSplit
(
int
num
,
int
numPartitions
)
{
if
(
numPartitions
<
1
||
num
<
0
||
num
>=
numPartitions
)
{
throw
new
IllegalArgumentException
();
...
...
@@ -36,5 +56,10 @@ public abstract class SplittableIterator<T> implements Iterator<T>, Serializable
return
split
(
numPartitions
)[
num
];
}
/**
* The maximum number of splits into which this iterator can be split up.
*
* @return The maximum number of splits into which this iterator can be split up.
*/
public
abstract
int
getMaximumNumberOfSplits
();
}
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录