Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
doujutun3207
flink
提交
63751058
F
flink
项目概览
doujutun3207
/
flink
与 Fork 源项目一致
从无法访问的项目Fork
通知
24
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
F
flink
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
63751058
编写于
7月 14, 2014
作者:
J
jfeher
提交者:
Stephan Ewen
8月 18, 2014
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
[streaming] RMQSink and Test
上级
04dbc25f
变更
6
显示空白变更内容
内联
并排
Showing
6 changed file
with
196 addition
and
6 deletion
+196
-6
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/performance/WordCountPerformanceSplitter.java
...e/streaming/performance/WordCountPerformanceSplitter.java
+10
-1
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/rabbitmq/RMQSink.java
...main/java/eu/stratosphere/streaming/rabbitmq/RMQSink.java
+90
-0
flink-addons/flink-streaming/src/test/java/eu/stratosphere/streaming/api/MapTest.java
.../src/test/java/eu/stratosphere/streaming/api/MapTest.java
+5
-0
flink-addons/flink-streaming/src/test/java/eu/stratosphere/streaming/api/RMQTest.java
.../src/test/java/eu/stratosphere/streaming/api/RMQTest.java
+75
-0
flink-addons/flink-streaming/src/test/java/eu/stratosphere/streaming/api/StreamCollector2Test.java
...a/eu/stratosphere/streaming/api/StreamCollector2Test.java
+3
-1
flink-addons/flink-streaming/src/test/java/eu/stratosphere/streaming/api/StreamCollectorTest.java
...va/eu/stratosphere/streaming/api/StreamCollectorTest.java
+13
-4
未找到文件。
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/performance/WordCountPerformanceSplitter.java
浏览文件 @
63751058
...
...
@@ -28,16 +28,25 @@ public class WordCountPerformanceSplitter extends FlatMapFunction<Tuple1<String>
PerformanceCounter
pCounter
=
new
PerformanceCounter
(
"SplitterEmitCounter"
,
1000
,
1000
,
30000
,
"/home/
mbalassi/strato-perf
.csv"
);
"/home/
judit/strato/perf/broadcast4
.csv"
);
@Override
public
void
flatMap
(
Tuple1
<
String
>
inTuple
,
Collector
<
Tuple1
<
String
>>
out
)
throws
Exception
{
for
(
String
word
:
inTuple
.
f0
.
split
(
" "
))
{
outTuple
.
f0
=
word
;
// pTimer.startTimer();
out
.
collect
(
outTuple
);
// pTimer.stopTimer();
pCounter
.
count
();
}
}
// @Override
// public String getResult() {
// pCounter.writeCSV();
// pTimer.writeCSV();
// return "";
// }
}
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/rabbitmq/RMQSink.java
0 → 100644
浏览文件 @
63751058
/***********************************************************************************************************************
*
* Copyright (C) 2010-2014 by the Stratosphere project (http://stratosphere.eu)
*
* Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
* an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
* specific language governing permissions and limitations under the License.
*
**********************************************************************************************************************/
package
eu.stratosphere.streaming.rabbitmq
;
import
java.io.IOException
;
import
eu.stratosphere.api.java.tuple.Tuple1
;
import
eu.stratosphere.streaming.api.SinkFunction
;
import
com.rabbitmq.client.ConnectionFactory
;
import
com.rabbitmq.client.Connection
;
import
com.rabbitmq.client.Channel
;
/**
* Source for sending messages to a RabbitMQ queue. The source currently only
* support string messages. Other types will be added soon.
*
*/
public
class
RMQSink
extends
SinkFunction
<
Tuple1
<
String
>>{
private
static
final
long
serialVersionUID
=
1L
;
private
String
QUEUE_NAME
;
private
String
HOST_NAME
;
private
transient
ConnectionFactory
factory
;
private
transient
Connection
connection
;
private
transient
Channel
channel
;
public
RMQSink
(
String
HOST_NAME
,
String
QUEUE_NAME
)
{
this
.
HOST_NAME
=
HOST_NAME
;
this
.
QUEUE_NAME
=
QUEUE_NAME
;
}
public
void
initializeConnection
(){
factory
=
new
ConnectionFactory
();
factory
.
setHost
(
HOST_NAME
);
try
{
connection
=
factory
.
newConnection
();
channel
=
connection
.
createChannel
();
}
catch
(
IOException
e
)
{
// TODO Auto-generated catch block
e
.
printStackTrace
();
}
}
@Override
public
void
invoke
(
Tuple1
<
String
>
tuple
)
{
initializeConnection
();
try
{
channel
.
queueDeclare
(
QUEUE_NAME
,
false
,
false
,
false
,
null
);
String
message
=
tuple
.
f0
;
channel
.
basicPublish
(
""
,
QUEUE_NAME
,
null
,
message
.
getBytes
());
System
.
out
.
println
(
" [x] Sent '"
+
message
+
"'"
);
}
catch
(
IOException
e1
)
{
// TODO Auto-generated catch block
e1
.
printStackTrace
();
}
try
{
channel
.
close
();
connection
.
close
();
}
catch
(
IOException
e
)
{
// TODO Auto-generated catch block
e
.
printStackTrace
();
}
}
}
flink-addons/flink-streaming/src/test/java/eu/stratosphere/streaming/api/MapTest.java
浏览文件 @
63751058
...
...
@@ -46,6 +46,7 @@ public class MapTest {
@Override
public
void
invoke
(
Collector
<
Tuple1
<
Integer
>>
collector
)
throws
Exception
{
for
(
int
i
=
0
;
i
<
5
;
i
++)
{
System
.
out
.
println
(
"s1: "
+
i
);
collector
.
collect
(
new
Tuple1
<
Integer
>(
i
));
}
}
...
...
@@ -56,6 +57,7 @@ public class MapTest {
@Override
public
void
invoke
(
Collector
<
Tuple1
<
Integer
>>
collector
)
throws
Exception
{
for
(
int
i
=
5
;
i
<
10
;
i
++)
{
System
.
out
.
println
(
"s2: "
+
i
);
collector
.
collect
(
new
Tuple1
<
Integer
>(
i
));
}
}
...
...
@@ -66,6 +68,7 @@ public class MapTest {
@Override
public
void
invoke
(
Collector
<
Tuple1
<
Integer
>>
collector
)
throws
Exception
{
for
(
int
i
=
10
;
i
<
15
;
i
++)
{
System
.
out
.
println
(
"s3: "
+
i
);
collector
.
collect
(
new
Tuple1
<
Integer
>(
i
));
}
}
...
...
@@ -85,6 +88,7 @@ public class MapTest {
@Override
public
Tuple1
<
Integer
>
map
(
Tuple1
<
Integer
>
value
)
throws
Exception
{
joinSetResult
.
add
(
value
.
f0
);
System
.
out
.
println
(
value
.
f0
);
return
new
Tuple1
<
Integer
>(
value
.
f0
);
}
}
...
...
@@ -167,6 +171,7 @@ public class MapTest {
@Override
public
void
invoke
(
Tuple1
<
Integer
>
tuple
)
{
System
.
out
.
println
(
"doing nothing"
);
}
}
...
...
flink-addons/flink-streaming/src/test/java/eu/stratosphere/streaming/api/RMQTest.java
0 → 100644
浏览文件 @
63751058
/***********************************************************************************************************************
*
* Copyright (C) 2010-2014 by the Stratosphere project (http://stratosphere.eu)
*
* Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
* an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
* specific language governing permissions and limitations under the License.
*
**********************************************************************************************************************/
package
eu.stratosphere.streaming.api
;
import
static
org
.
junit
.
Assert
.*;
import
java.util.HashSet
;
import
java.util.Set
;
import
org.junit.Test
;
import
eu.stratosphere.api.java.functions.MapFunction
;
import
eu.stratosphere.api.java.tuple.Tuple1
;
import
eu.stratosphere.streaming.rabbitmq.RMQSink
;
import
eu.stratosphere.streaming.rabbitmq.RMQSource
;
import
eu.stratosphere.util.Collector
;
public
class
RMQTest
{
public
static
final
class
MySink
extends
SinkFunction
<
Tuple1
<
String
>>
{
@Override
public
void
invoke
(
Tuple1
<
String
>
tuple
)
{
result
.
add
(
tuple
.
f0
);
}
}
private
static
Set
<
String
>
expected
=
new
HashSet
<
String
>();
private
static
Set
<
String
>
result
=
new
HashSet
<
String
>();
private
static
void
fillExpected
()
{
expected
.
add
(
"one"
);
expected
.
add
(
"two"
);
expected
.
add
(
"three"
);
expected
.
add
(
"four"
);
expected
.
add
(
"five"
);
}
@Test
public
void
RMQTest1
()
throws
Exception
{
StreamExecutionEnvironment
env
=
new
StreamExecutionEnvironment
();
DataStream
<
Tuple1
<
String
>>
dataStream1
=
env
.
addSource
(
new
RMQSource
(
"localhost"
,
"hello"
),
1
)
.
addSink
(
new
MySink
());
DataStream
<
Tuple1
<
String
>>
dataStream2
=
env
.
fromElements
(
"one"
,
"two"
,
"three"
,
"four"
,
"five"
,
"q"
)
.
addSink
(
new
RMQSink
(
"localhost"
,
"hello"
));
env
.
execute
();
fillExpected
();
assertEquals
(
expected
,
result
);
}
}
flink-addons/flink-streaming/src/test/java/eu/stratosphere/streaming/api/StreamCollector
Manager
Test.java
→
flink-addons/flink-streaming/src/test/java/eu/stratosphere/streaming/api/StreamCollector
2
Test.java
浏览文件 @
63751058
...
...
@@ -29,7 +29,7 @@ import eu.stratosphere.streaming.api.streamcomponent.MockRecordWriter;
import
eu.stratosphere.streaming.api.streamrecord.StreamRecord
;
import
eu.stratosphere.streaming.util.MockRecordWriterFactory
;
public
class
StreamCollector
Manager
Test
{
public
class
StreamCollector
2
Test
{
StreamCollectorManager
<
Tuple
>
collector
;
...
...
@@ -55,7 +55,9 @@ public class StreamCollectorManagerTest {
fOut
.
add
(
rw2
);
collector
=
new
StreamCollectorManager
<
Tuple
>(
batchSizesOfNotPartitioned
,
batchSizesOfPartitioned
,
parallelismOfOutput
,
keyPosition
,
batchTimeout
,
channelID
,
null
,
fOut
,
fOut
);
Tuple1
<
Integer
>
t
=
new
Tuple1
<
Integer
>();
StreamCollector
<
Tuple
>
sc1
=
new
StreamCollector
<
Tuple
>(
1
,
batchTimeout
,
channelID
,
null
);
t
.
f0
=
0
;
collector
.
collect
(
t
);
...
...
flink-addons/flink-streaming/src/test/java/eu/stratosphere/streaming/api/StreamCollectorTest.java
浏览文件 @
63751058
...
...
@@ -16,24 +16,31 @@
package
eu.stratosphere.streaming.api
;
import
static
org
.
junit
.
Assert
.
assertEquals
;
import
static
org
.
mockito
.
Mockito
.
doCallRealMethod
;
import
static
org
.
mockito
.
Mockito
.
mock
;
import
java.util.ArrayList
;
import
org.junit.Test
;
import
org.mockito.Mockito
;
import
eu.stratosphere.api.java.tuple.Tuple1
;
import
eu.stratosphere.nephele.io.RecordWriter
;
import
eu.stratosphere.streaming.api.streamcomponent.MockRecordWriter
;
import
eu.stratosphere.streaming.api.streamrecord.StreamRecord
;
import
eu.stratosphere.streaming.util.MockRecordWriterFactory
;
public
class
StreamCollectorTest
{
@Test
public
void
testStreamCollector
()
{
StreamCollector
<
Tuple1
<
Integer
>>
collector
=
new
StreamCollector
<
Tuple1
<
Integer
>>
(
10
,
1000
,
0
,
null
);
StreamCollector
collector
=
new
StreamCollector
(
10
,
1000
,
0
,
null
);
assertEquals
(
10
,
collector
.
batchSize
);
}
@Test
public
void
testCollect
()
{
StreamCollector
<
Tuple1
<
Integer
>>
collector
=
new
StreamCollector
<
Tuple1
<
Integer
>>
(
2
,
1000
,
0
,
null
);
StreamCollector
collector
=
new
StreamCollector
(
2
,
1000
,
0
,
null
);
collector
.
collect
(
new
Tuple1
<
Integer
>(
3
));
collector
.
collect
(
new
Tuple1
<
Integer
>(
4
));
collector
.
collect
(
new
Tuple1
<
Integer
>(
5
));
...
...
@@ -43,7 +50,8 @@ public class StreamCollectorTest {
@Test
public
void
testBatchSize
()
throws
InterruptedException
{
StreamCollector
<
Tuple1
<
Integer
>>
collector
=
new
StreamCollector
<
Tuple1
<
Integer
>>(
3
,
100
,
0
,
null
);
System
.
out
.
println
(
"---------------"
);
StreamCollector
collector
=
new
StreamCollector
(
3
,
100
,
0
,
null
);
collector
.
collect
(
new
Tuple1
<
Integer
>(
0
));
collector
.
collect
(
new
Tuple1
<
Integer
>(
0
));
collector
.
collect
(
new
Tuple1
<
Integer
>(
0
));
...
...
@@ -51,13 +59,14 @@ public class StreamCollectorTest {
Thread
.
sleep
(
200
);
collector
.
collect
(
new
Tuple1
<
Integer
>(
2
));
collector
.
collect
(
new
Tuple1
<
Integer
>(
3
));
System
.
out
.
println
(
"---------------"
);
}
@Test
public
void
recordWriter
()
{
MockRecordWriter
recWriter
=
MockRecordWriterFactory
.
create
();
StreamCollector
<
Tuple1
<
Integer
>>
collector
=
new
StreamCollector
<
Tuple1
<
Integer
>>
(
2
,
1000
,
0
,
null
,
recWriter
);
StreamCollector
collector
=
new
StreamCollector
(
2
,
1000
,
0
,
null
,
recWriter
);
collector
.
collect
(
new
Tuple1
<
Integer
>(
3
));
collector
.
collect
(
new
Tuple1
<
Integer
>(
4
));
collector
.
collect
(
new
Tuple1
<
Integer
>(
5
));
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录