Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
doujutun3207
flink
提交
e547f4f8
F
flink
项目概览
doujutun3207
/
flink
与 Fork 源项目一致
从无法访问的项目Fork
通知
24
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
F
flink
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
e547f4f8
编写于
7月 14, 2014
作者:
G
gaborhermann
提交者:
Stephan Ewen
8月 18, 2014
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
[streaming] Updated input checking for next record at invoke
上级
fe142630
变更
5
隐藏空白更改
内联
并排
Showing
5 changed file
with
174 addition
and
16 deletion
+174
-16
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/api/streamcomponent/StreamComponentHelper.java
.../streaming/api/streamcomponent/StreamComponentHelper.java
+13
-8
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/api/streamcomponent/StreamRecordReader.java
...ere/streaming/api/streamcomponent/StreamRecordReader.java
+155
-0
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/api/streamcomponent/StreamSink.java
...tratosphere/streaming/api/streamcomponent/StreamSink.java
+2
-3
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/api/streamcomponent/StreamTask.java
...tratosphere/streaming/api/streamcomponent/StreamTask.java
+2
-3
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/examples/wordcount/WordCountLocal.java
...tosphere/streaming/examples/wordcount/WordCountLocal.java
+2
-2
未找到文件。
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/api/streamcomponent/StreamComponentHelper.java
浏览文件 @
e547f4f8
...
...
@@ -17,6 +17,7 @@ package eu.stratosphere.streaming.api.streamcomponent;
import
java.io.IOException
;
import
java.util.ConcurrentModificationException
;
import
java.util.LinkedList
;
import
java.util.List
;
import
org.apache.commons.logging.Log
;
...
...
@@ -26,7 +27,6 @@ import eu.stratosphere.configuration.Configuration;
import
eu.stratosphere.nephele.event.task.AbstractTaskEvent
;
import
eu.stratosphere.nephele.event.task.EventListener
;
import
eu.stratosphere.nephele.io.ChannelSelector
;
import
eu.stratosphere.nephele.io.RecordReader
;
import
eu.stratosphere.nephele.io.RecordWriter
;
import
eu.stratosphere.nephele.template.AbstractInvokable
;
import
eu.stratosphere.streaming.api.invokable.DefaultSinkInvokable
;
...
...
@@ -75,15 +75,15 @@ public final class StreamComponentHelper<T extends AbstractInvokable> {
}
public
void
setConfigInputs
(
T
taskBase
,
Configuration
taskConfiguration
,
List
<
RecordReader
<
StreamRecord
>>
inputs
)
public
void
setConfigInputs
(
T
taskBase
,
Configuration
taskConfiguration
,
List
<
Stream
RecordReader
<
StreamRecord
>>
inputs
)
throws
StreamComponentException
{
int
numberOfInputs
=
taskConfiguration
.
getInteger
(
"numberOfInputs"
,
0
);
for
(
int
i
=
0
;
i
<
numberOfInputs
;
i
++)
{
if
(
taskBase
instanceof
StreamTask
)
{
inputs
.
add
(
new
RecordReader
<
StreamRecord
>((
StreamTask
)
taskBase
,
StreamRecord
.
class
));
inputs
.
add
(
new
Stream
RecordReader
<
StreamRecord
>((
StreamTask
)
taskBase
,
StreamRecord
.
class
));
}
else
if
(
taskBase
instanceof
StreamSink
)
{
inputs
.
add
(
new
RecordReader
<
StreamRecord
>((
StreamSink
)
taskBase
,
StreamRecord
.
class
));
inputs
.
add
(
new
Stream
RecordReader
<
StreamRecord
>((
StreamSink
)
taskBase
,
StreamRecord
.
class
));
}
else
{
throw
new
StreamComponentException
(
"Nonsupported object passed to setConfigInputs"
);
}
...
...
@@ -142,7 +142,7 @@ public final class StreamComponentHelper<T extends AbstractInvokable> {
}
// TODO find a better solution for this
public
void
threadSafePublish
(
AbstractTaskEvent
event
,
RecordReader
<
StreamRecord
>
input
)
public
void
threadSafePublish
(
AbstractTaskEvent
event
,
Stream
RecordReader
<
StreamRecord
>
input
)
throws
InterruptedException
,
IOException
{
boolean
concurrentModificationOccured
=
false
;
...
...
@@ -177,12 +177,13 @@ public final class StreamComponentHelper<T extends AbstractInvokable> {
}
}
public
void
invokeRecords
(
RecordInvokable
userFunction
,
List
<
RecordReader
<
StreamRecord
>>
inputs
,
String
name
)
throws
Exception
{
public
void
invokeRecords
(
RecordInvokable
userFunction
,
List
<
StreamRecordReader
<
StreamRecord
>>
inputs
,
String
name
)
throws
Exception
{
List
<
StreamRecordReader
<
StreamRecord
>>
closedInputs
=
new
LinkedList
<
StreamRecordReader
<
StreamRecord
>>();
boolean
hasInput
=
true
;
while
(
hasInput
)
{
hasInput
=
false
;
for
(
RecordReader
<
StreamRecord
>
input
:
inputs
)
{
for
(
Stream
RecordReader
<
StreamRecord
>
input
:
inputs
)
{
if
(
input
.
hasNext
())
{
hasInput
=
true
;
StreamRecord
record
=
input
.
next
();
...
...
@@ -191,7 +192,11 @@ public final class StreamComponentHelper<T extends AbstractInvokable> {
threadSafePublish
(
new
AckEvent
(
id
),
input
);
log
.
debug
(
"ACK: "
+
id
+
" -- "
+
name
);
}
else
if
(
input
.
isInputClosed
())
{
closedInputs
.
add
(
input
);
}
}
inputs
.
removeAll
(
closedInputs
);
}
}
...
...
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/api/streamcomponent/StreamRecordReader.java
0 → 100644
浏览文件 @
e547f4f8
/***********************************************************************************************************************
* Copyright (C) 2010-2013 by the Stratosphere project (http://stratosphere.eu)
*
* Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
* an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
* specific language governing permissions and limitations under the License.
**********************************************************************************************************************/
package
eu.stratosphere.streaming.api.streamcomponent
;
import
java.io.IOException
;
import
eu.stratosphere.core.io.IOReadableWritable
;
import
eu.stratosphere.nephele.io.AbstractSingleGateRecordReader
;
import
eu.stratosphere.nephele.io.InputChannelResult
;
import
eu.stratosphere.nephele.io.MutableRecordDeserializerFactory
;
import
eu.stratosphere.nephele.io.Reader
;
import
eu.stratosphere.nephele.template.AbstractOutputTask
;
import
eu.stratosphere.nephele.template.AbstractTask
;
/**
* A record writer connects an input gate to an application. It allows the application
* query for incoming records and read them from input gate.
*
* @param <T> The type of the record that can be read from this record reader.
*/
public
class
StreamRecordReader
<
T
extends
IOReadableWritable
>
extends
AbstractSingleGateRecordReader
<
T
>
implements
Reader
<
T
>
{
private
final
Class
<
T
>
recordType
;
/**
* Stores the last read record.
*/
private
T
lookahead
;
/**
* Stores if more no more records will be received from the assigned input gate.
*/
private
boolean
noMoreRecordsWillFollow
;
// --------------------------------------------------------------------------------------------
/**
* Constructs a new record reader and registers a new input gate with the application's environment.
*
* @param taskBase
* The application that instantiated the record reader.
* @param recordType
* The class of records that can be read from the record reader.
*/
public
StreamRecordReader
(
AbstractTask
taskBase
,
Class
<
T
>
recordType
)
{
super
(
taskBase
,
MutableRecordDeserializerFactory
.<
T
>
get
(),
0
);
this
.
recordType
=
recordType
;
}
/**
* Constructs a new record reader and registers a new input gate with the application's environment.
*
* @param outputBase
* The application that instantiated the record reader.
* @param recordType
* The class of records that can be read from the record reader.
*/
public
StreamRecordReader
(
AbstractOutputTask
outputBase
,
Class
<
T
>
recordType
)
{
super
(
outputBase
,
MutableRecordDeserializerFactory
.<
T
>
get
(),
0
);
this
.
recordType
=
recordType
;
}
// --------------------------------------------------------------------------------------------
/**
* Checks if at least one more record can be read from the associated input gate. This method may block
* until the associated input gate is able to read the record from one of its input channels.
*
* @return <code>true</code>it at least one more record can be read from the associated input gate, otherwise
* <code>false</code>
*/
@Override
public
boolean
hasNext
()
throws
IOException
,
InterruptedException
{
if
(
this
.
lookahead
!=
null
)
{
return
true
;
}
else
{
if
(
this
.
noMoreRecordsWillFollow
)
{
return
false
;
}
T
record
=
instantiateRecordType
();
while
(
true
)
{
InputChannelResult
result
=
this
.
inputGate
.
readRecord
(
record
);
switch
(
result
)
{
case
INTERMEDIATE_RECORD_FROM_BUFFER:
case
LAST_RECORD_FROM_BUFFER:
this
.
lookahead
=
record
;
return
true
;
case
END_OF_SUPERSTEP:
if
(
incrementEndOfSuperstepEventAndCheck
())
return
false
;
else
return
false
;
case
TASK_EVENT:
handleEvent
(
this
.
inputGate
.
getCurrentEvent
());
return
false
;
case
END_OF_STREAM:
this
.
noMoreRecordsWillFollow
=
true
;
return
false
;
case
NONE:
// internal event or an incomplete record that needs further chunks
// the current unit is exhausted
break
;
default
:
return
false
;
}
}
}
}
/**
* Reads the current record from the associated input gate.
*
* @return the current record from the associated input gate.
* @throws IOException
* thrown if any error occurs while reading the record from the input gate
*/
@Override
public
T
next
()
throws
IOException
,
InterruptedException
{
if
(
hasNext
())
{
T
tmp
=
this
.
lookahead
;
this
.
lookahead
=
null
;
return
tmp
;
}
else
{
return
null
;
}
}
@Override
public
boolean
isInputClosed
()
{
return
this
.
noMoreRecordsWillFollow
;
}
private
T
instantiateRecordType
()
{
try
{
return
this
.
recordType
.
newInstance
();
}
catch
(
InstantiationException
e
)
{
throw
new
RuntimeException
(
"Cannot instantiate class '"
+
this
.
recordType
.
getName
()
+
"'."
,
e
);
}
catch
(
IllegalAccessException
e
)
{
throw
new
RuntimeException
(
"Cannot instantiate class '"
+
this
.
recordType
.
getName
()
+
"'."
,
e
);
}
}
}
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/api/streamcomponent/StreamSink.java
浏览文件 @
e547f4f8
...
...
@@ -22,7 +22,6 @@ import org.apache.commons.logging.Log;
import
org.apache.commons.logging.LogFactory
;
import
eu.stratosphere.configuration.Configuration
;
import
eu.stratosphere.nephele.io.RecordReader
;
import
eu.stratosphere.nephele.template.AbstractOutputTask
;
import
eu.stratosphere.streaming.api.invokable.UserSinkInvokable
;
import
eu.stratosphere.streaming.api.streamrecord.StreamRecord
;
...
...
@@ -31,14 +30,14 @@ public class StreamSink extends AbstractOutputTask {
private
static
final
Log
log
=
LogFactory
.
getLog
(
StreamSink
.
class
);
private
List
<
RecordReader
<
StreamRecord
>>
inputs
;
private
List
<
Stream
RecordReader
<
StreamRecord
>>
inputs
;
private
UserSinkInvokable
userFunction
;
private
StreamComponentHelper
<
StreamSink
>
streamSinkHelper
;
private
String
name
;
public
StreamSink
()
{
// TODO: Make configuration file visible and call setClassInputs() here
inputs
=
new
LinkedList
<
RecordReader
<
StreamRecord
>>();
inputs
=
new
LinkedList
<
Stream
RecordReader
<
StreamRecord
>>();
userFunction
=
null
;
streamSinkHelper
=
new
StreamComponentHelper
<
StreamSink
>();
}
...
...
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/api/streamcomponent/StreamTask.java
浏览文件 @
e547f4f8
...
...
@@ -23,7 +23,6 @@ import org.apache.commons.logging.LogFactory;
import
eu.stratosphere.configuration.Configuration
;
import
eu.stratosphere.nephele.io.ChannelSelector
;
import
eu.stratosphere.nephele.io.RecordReader
;
import
eu.stratosphere.nephele.io.RecordWriter
;
import
eu.stratosphere.nephele.template.AbstractTask
;
import
eu.stratosphere.streaming.api.invokable.UserTaskInvokable
;
...
...
@@ -34,7 +33,7 @@ public class StreamTask extends AbstractTask {
private
static
final
Log
log
=
LogFactory
.
getLog
(
StreamTask
.
class
);
private
List
<
RecordReader
<
StreamRecord
>>
inputs
;
private
List
<
Stream
RecordReader
<
StreamRecord
>>
inputs
;
private
List
<
RecordWriter
<
StreamRecord
>>
outputs
;
private
List
<
ChannelSelector
<
StreamRecord
>>
partitioners
;
private
UserTaskInvokable
userFunction
;
...
...
@@ -48,7 +47,7 @@ public class StreamTask extends AbstractTask {
public
StreamTask
()
{
// TODO: Make configuration file visible and call setClassInputs() here
inputs
=
new
LinkedList
<
RecordReader
<
StreamRecord
>>();
inputs
=
new
LinkedList
<
Stream
RecordReader
<
StreamRecord
>>();
outputs
=
new
LinkedList
<
RecordWriter
<
StreamRecord
>>();
partitioners
=
new
LinkedList
<
ChannelSelector
<
StreamRecord
>>();
userFunction
=
null
;
...
...
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/examples/wordcount/WordCountLocal.java
浏览文件 @
e547f4f8
...
...
@@ -31,8 +31,8 @@ public class WordCountLocal {
private
static
JobGraph
getJobGraph
()
throws
Exception
{
JobGraphBuilder
graphBuilder
=
new
JobGraphBuilder
(
"testGraph"
);
graphBuilder
.
setSource
(
"WordCountSource"
,
WordCountDummySource
.
class
);
graphBuilder
.
setTask
(
"WordCountSplitter"
,
WordCountSplitter
.
class
,
2
,
1
);
graphBuilder
.
setTask
(
"WordCountCounter"
,
WordCountCounter
.
class
,
2
,
1
);
graphBuilder
.
setTask
(
"WordCountSplitter"
,
WordCountSplitter
.
class
,
2
,
2
);
graphBuilder
.
setTask
(
"WordCountCounter"
,
WordCountCounter
.
class
,
2
,
2
);
graphBuilder
.
setSink
(
"WordCountSink"
,
WordCountSink
.
class
);
graphBuilder
.
shuffleConnect
(
"WordCountSource"
,
"WordCountSplitter"
);
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录