Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
doujutun3207
flink
提交
b5dfc948
F
flink
项目概览
doujutun3207
/
flink
与 Fork 源项目一致
从无法访问的项目Fork
通知
24
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
F
flink
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
b5dfc948
编写于
7月 14, 2014
作者:
Y
Yingjun Wu
提交者:
Stephan Ewen
8月 18, 2014
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
[streaming] update iterative jobs
上级
c92727b8
变更
25
隐藏空白更改
内联
并排
Showing
25 changed file
with
686 addition
and
203 deletion
+686
-203
flink-addons/flink-streaming/pom.xml
flink-addons/flink-streaming/pom.xml
+6
-1
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/examples/iterative/collaborativefilter/CollaborativeFilteringLocal.java
...tive/collaborativefilter/CollaborativeFilteringLocal.java
+23
-2
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/examples/iterative/collaborativefilter/CollaborativeFilteringSink.java
...ative/collaborativefilter/CollaborativeFilteringSink.java
+14
-6
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/examples/iterative/collaborativefilter/CollaborativeFilteringSource.java
...ive/collaborativefilter/CollaborativeFilteringSource.java
+54
-0
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/examples/iterative/collaborativefilter/CollaborativeFilteringTask.java
...ative/collaborativefilter/CollaborativeFilteringTask.java
+52
-0
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/examples/iterative/kmeans/KMeansLocal.java
...here/streaming/examples/iterative/kmeans/KMeansLocal.java
+27
-2
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/examples/iterative/kmeans/KMeansSink.java
...phere/streaming/examples/iterative/kmeans/KMeansSink.java
+6
-3
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/examples/iterative/kmeans/KMeansSource.java
...ere/streaming/examples/iterative/kmeans/KMeansSource.java
+84
-0
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/examples/iterative/kmeans/KMeansTask.java
...phere/streaming/examples/iterative/kmeans/KMeansTask.java
+42
-0
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/examples/iterative/pagerank/Graph.java
...tosphere/streaming/examples/iterative/pagerank/Graph.java
+47
-0
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/examples/iterative/pagerank/PageRankLocal.java
.../streaming/examples/iterative/pagerank/PageRankLocal.java
+46
-0
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/examples/iterative/pagerank/PageRankSink.java
...e/streaming/examples/iterative/pagerank/PageRankSink.java
+37
-0
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/examples/iterative/pagerank/PageRankSource.java
...streaming/examples/iterative/pagerank/PageRankSource.java
+52
-0
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/examples/iterative/pagerank/PageRankTask.java
...e/streaming/examples/iterative/pagerank/PageRankTask.java
+17
-3
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/examples/iterative/sssp/Graph.java
...stratosphere/streaming/examples/iterative/sssp/Graph.java
+47
-0
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/examples/iterative/sssp/SSSPLocal.java
...tosphere/streaming/examples/iterative/sssp/SSSPLocal.java
+23
-2
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/examples/iterative/sssp/SSSPSink.java
...atosphere/streaming/examples/iterative/sssp/SSSPSink.java
+37
-0
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/examples/iterative/sssp/SSSPSource.java
...osphere/streaming/examples/iterative/sssp/SSSPSource.java
+52
-0
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/examples/iterative/sssp/SSSPTask.java
...atosphere/streaming/examples/iterative/sssp/SSSPTask.java
+11
-3
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/examples/join/JoinLocal.java
...va/eu/stratosphere/streaming/examples/join/JoinLocal.java
+2
-40
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/examples/window/join/WindowJoinLocal.java
...phere/streaming/examples/window/join/WindowJoinLocal.java
+2
-40
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/examples/window/sum/WindowSumLocal.java
...osphere/streaming/examples/window/sum/WindowSumLocal.java
+2
-40
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/examples/window/wordcount/WindowWordCountLocal.java
...aming/examples/window/wordcount/WindowWordCountLocal.java
+2
-40
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/examples/wordcount/WordCountLocal.java
...tosphere/streaming/examples/wordcount/WordCountLocal.java
+1
-12
flink-addons/flink-streaming/src/main/java/helloworld.java
flink-addons/flink-streaming/src/main/java/helloworld.java
+0
-9
未找到文件。
flink-addons/flink-streaming/pom.xml
浏览文件 @
b5dfc948
...
...
@@ -87,7 +87,12 @@
<dependency>
<groupId>
org.apache.kafka
</groupId>
<artifactId>
kafka_2.10
</artifactId>
<version>
0.8.1.1
</version>
<version>
0.8.0
</version>
</dependency>
<dependency>
<groupId>
org.jblas
</groupId>
<artifactId>
jblas
</artifactId>
<version>
1.2.3
</version>
</dependency>
</dependencies>
...
...
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/examples/iterative/collaborativefilter/CollaborativeFilteringLocal.java
浏览文件 @
b5dfc948
...
...
@@ -15,11 +15,32 @@
package
eu.stratosphere.streaming.examples.iterative.collaborativefilter
;
import
org.apache.log4j.Level
;
import
eu.stratosphere.nephele.jobgraph.JobGraph
;
import
eu.stratosphere.streaming.api.JobGraphBuilder
;
import
eu.stratosphere.streaming.faulttolerance.FaultToleranceType
;
import
eu.stratosphere.streaming.util.ClusterUtil
;
import
eu.stratosphere.streaming.util.LogUtils
;
public
class
CollaborativeFilteringLocal
{
public
static
JobGraph
getJobGraph
()
{
JobGraphBuilder
graphBuilder
=
new
JobGraphBuilder
(
"testGraph"
,
FaultToleranceType
.
NONE
);
graphBuilder
.
setSource
(
"Source"
,
new
CollaborativeFilteringSource
());
graphBuilder
.
setTask
(
"Task"
,
new
CollaborativeFilteringTask
(),
1
,
1
);
graphBuilder
.
setSink
(
"Sink"
,
new
CollaborativeFilteringSink
());
public
static
void
main
(
String
[]
args
)
{
// TODO Auto-generated method stub
graphBuilder
.
fieldsConnect
(
"Source"
,
"Task"
,
0
);
graphBuilder
.
shuffleConnect
(
"Task"
,
"Sink"
);
return
graphBuilder
.
getJobGraph
();
}
public
static
void
main
(
String
[]
args
)
{
LogUtils
.
initializeDefaultConsoleLogger
(
Level
.
DEBUG
,
Level
.
INFO
);
ClusterUtil
.
runOnMiniCluster
(
getJobGraph
());
}
}
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/examples/iterative/collaborativefilter/C
FSource
.java
→
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/examples/iterative/collaborativefilter/C
ollaborativeFilteringSink
.java
浏览文件 @
b5dfc948
...
...
@@ -15,15 +15,23 @@
package
eu.stratosphere.streaming.examples.iterative.collaborativefilter
;
import
eu.stratosphere.streaming.api.invokable.UserSourceInvokable
;
import
eu.stratosphere.streaming.api.invokable.UserSinkInvokable
;
import
eu.stratosphere.streaming.api.streamrecord.StreamRecord
;
public
class
C
FSource
extends
UserSource
Invokable
{
public
class
C
ollaborativeFilteringSink
extends
UserSink
Invokable
{
private
static
final
long
serialVersionUID
=
1L
;
@Override
public
void
invoke
()
throws
Exception
{
public
void
invoke
(
StreamRecord
record
)
throws
Exception
{
// TODO Auto-generated method stub
System
.
out
.
println
(
"received record..."
);
int
tupleNum
=
record
.
getNumOfTuples
();
System
.
out
.
println
(
"============================================"
);
for
(
int
i
=
0
;
i
<
tupleNum
;
++
i
)
{
System
.
out
.
println
(
"name="
+
record
.
getField
(
i
,
0
)
+
", grade="
+
record
.
getField
(
i
,
1
)
+
", salary="
+
record
.
getField
(
i
,
2
));
}
System
.
out
.
println
(
"============================================"
);
}
}
}
\ No newline at end of file
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/examples/iterative/collaborativefilter/CollaborativeFilteringSource.java
0 → 100644
浏览文件 @
b5dfc948
/***********************************************************************************************************************
*
* Copyright (C) 2010-2014 by the Stratosphere project (http://stratosphere.eu)
*
* Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
* an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
* specific language governing permissions and limitations under the License.
*
**********************************************************************************************************************/
package
eu.stratosphere.streaming.examples.iterative.collaborativefilter
;
import
java.io.BufferedReader
;
import
java.io.FileReader
;
import
eu.stratosphere.api.java.tuple.Tuple3
;
import
eu.stratosphere.streaming.api.invokable.UserSourceInvokable
;
import
eu.stratosphere.streaming.api.streamrecord.StreamRecord
;
public
class
CollaborativeFilteringSource
extends
UserSourceInvokable
{
private
static
final
long
serialVersionUID
=
1L
;
private
BufferedReader
br
=
null
;
private
String
line
=
new
String
();
private
StreamRecord
outRecord
=
new
StreamRecord
(
new
Tuple3
<
Integer
,
Integer
,
Integer
>());
@Override
public
void
invoke
()
throws
Exception
{
// TODO Auto-generated method stub
br
=
new
BufferedReader
(
new
FileReader
(
"src/test/resources/testdata/MovieLens100k.data"
));
while
(
true
)
{
line
=
br
.
readLine
();
if
(
line
==
null
)
{
break
;
}
if
(
line
!=
""
)
{
String
[]
items
=
line
.
split
(
"\t"
);
outRecord
.
setInteger
(
0
,
Integer
.
valueOf
(
items
[
0
]));
outRecord
.
setInteger
(
1
,
Integer
.
valueOf
(
items
[
1
]));
outRecord
.
setInteger
(
2
,
Integer
.
valueOf
(
items
[
2
]));
emit
(
outRecord
);
performanceCounter
.
count
();
}
line
=
br
.
readLine
();
}
}
}
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/examples/iterative/collaborativefilter/CollaborativeFilteringTask.java
0 → 100644
浏览文件 @
b5dfc948
/***********************************************************************************************************************
*
* Copyright (C) 2010-2014 by the Stratosphere project (http://stratosphere.eu)
*
* Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
* an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
* specific language governing permissions and limitations under the License.
*
**********************************************************************************************************************/
package
eu.stratosphere.streaming.examples.iterative.collaborativefilter
;
import
java.util.HashMap
;
import
org.jblas.DoubleMatrix
;
import
eu.stratosphere.api.java.tuple.Tuple1
;
import
eu.stratosphere.streaming.api.invokable.UserTaskInvokable
;
import
eu.stratosphere.streaming.api.streamrecord.StreamRecord
;
public
class
CollaborativeFilteringTask
extends
UserTaskInvokable
{
private
static
final
long
serialVersionUID
=
1L
;
private
StreamRecord
outRecord
=
new
StreamRecord
(
new
Tuple1
<
String
>());
HashMap
<
Integer
,
Integer
>
rowIndex
=
new
HashMap
<
Integer
,
Integer
>();
HashMap
<
Integer
,
Integer
>
columnIndex
=
new
HashMap
<
Integer
,
Integer
>();
DoubleMatrix
userItem
=
new
DoubleMatrix
(
1000
,
2000
);
DoubleMatrix
coOccurence
=
new
DoubleMatrix
(
2000
,
2000
);
@Override
public
void
invoke
(
StreamRecord
record
)
throws
Exception
{
// TODO Auto-generated method stub
int
userId
=
record
.
getInteger
(
0
,
0
);
int
itemId
=
record
.
getInteger
(
0
,
1
);
int
rating
=
record
.
getInteger
(
0
,
2
);
if
(!
rowIndex
.
containsKey
(
userId
)){
rowIndex
.
put
(
userId
,
rowIndex
.
size
());
}
if
(!
columnIndex
.
containsKey
(
itemId
)){
columnIndex
.
put
(
itemId
,
columnIndex
.
size
());
}
userItem
.
put
(
rowIndex
.
get
(
userId
),
columnIndex
.
get
(
itemId
),
rating
);
//outRecord.setString(0, line);
}
}
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/examples/iterative/kmeans/KMeansLocal.java
浏览文件 @
b5dfc948
...
...
@@ -15,11 +15,36 @@
package
eu.stratosphere.streaming.examples.iterative.kmeans
;
import
org.apache.log4j.Level
;
import
eu.stratosphere.nephele.jobgraph.JobGraph
;
import
eu.stratosphere.streaming.api.JobGraphBuilder
;
import
eu.stratosphere.streaming.faulttolerance.FaultToleranceType
;
import
eu.stratosphere.streaming.util.ClusterUtil
;
import
eu.stratosphere.streaming.util.LogUtils
;
public
class
KMeansLocal
{
public
static
JobGraph
getJobGraph
()
{
int
numCenter
=
3
;
int
dimension
=
2
;
double
stddev
=
0.08
;
double
range
=
100.0
;
JobGraphBuilder
graphBuilder
=
new
JobGraphBuilder
(
"testGraph"
,
FaultToleranceType
.
NONE
);
graphBuilder
.
setSource
(
"Source"
,
new
KMeansSource
(
numCenter
,
dimension
,
stddev
,
range
));
graphBuilder
.
setTask
(
"Task"
,
new
KMeansTask
(
dimension
),
1
,
1
);
graphBuilder
.
setSink
(
"Sink"
,
new
KMeansSink
());
public
static
void
main
(
String
[]
args
)
{
// TODO Auto-generated method stub
graphBuilder
.
fieldsConnect
(
"Source"
,
"Task"
,
0
);
graphBuilder
.
shuffleConnect
(
"Task"
,
"Sink"
);
return
graphBuilder
.
getJobGraph
();
}
public
static
void
main
(
String
[]
args
)
{
LogUtils
.
initializeDefaultConsoleLogger
(
Level
.
DEBUG
,
Level
.
INFO
);
ClusterUtil
.
runOnMiniCluster
(
getJobGraph
());
}
}
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/examples/iterative/
collaborativefilter/CF
Sink.java
→
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/examples/iterative/
kmeans/KMeans
Sink.java
浏览文件 @
b5dfc948
...
...
@@ -13,17 +13,20 @@
*
**********************************************************************************************************************/
package
eu.stratosphere.streaming.examples.iterative.
collaborativefilter
;
package
eu.stratosphere.streaming.examples.iterative.
kmeans
;
import
eu.stratosphere.streaming.api.invokable.UserSinkInvokable
;
import
eu.stratosphere.streaming.api.streamrecord.StreamRecord
;
public
class
CF
Sink
extends
UserSinkInvokable
{
public
class
KMeans
Sink
extends
UserSinkInvokable
{
private
static
final
long
serialVersionUID
=
1L
;
@Override
public
void
invoke
(
StreamRecord
record
)
throws
Exception
{
// TODO Auto-generated method stub
//int tupleNum = record.getNumOfTuples();
System
.
out
.
println
(
"============================================"
);
System
.
out
.
println
(
"record="
+
record
.
getString
(
0
,
0
));
System
.
out
.
println
(
"============================================"
);
}
}
\ No newline at end of file
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/examples/iterative/kmeans/KMeansSource.java
0 → 100644
浏览文件 @
b5dfc948
/***********************************************************************************************************************
*
* Copyright (C) 2010-2014 by the Stratosphere project (http://stratosphere.eu)
*
* Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
* an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
* specific language governing permissions and limitations under the License.
*
**********************************************************************************************************************/
package
eu.stratosphere.streaming.examples.iterative.kmeans
;
import
java.util.Random
;
import
eu.stratosphere.api.java.tuple.Tuple1
;
import
eu.stratosphere.streaming.api.invokable.UserSourceInvokable
;
import
eu.stratosphere.streaming.api.streamrecord.StreamRecord
;
public
class
KMeansSource
extends
UserSourceInvokable
{
private
static
final
long
serialVersionUID
=
1L
;
private
static
final
long
DEFAULT_SEED
=
4650285087650871364L
;
private
Random
random
=
new
Random
(
DEFAULT_SEED
);
private
StreamRecord
outRecord
=
new
StreamRecord
(
new
Tuple1
<
String
>());
private
int
numCenter
;
private
int
dimension
;
private
double
absoluteStdDev
;
private
double
range
;
private
StringBuilder
buffer
=
new
StringBuilder
();
public
KMeansSource
(
int
numCenter
,
int
dimension
,
double
stddev
,
double
range
){
this
.
numCenter
=
numCenter
;
this
.
dimension
=
dimension
;
this
.
absoluteStdDev
=
stddev
*
range
;
this
.
range
=
range
;
}
@Override
public
void
invoke
()
throws
Exception
{
// TODO Auto-generated method stub
double
[][]
means
=
uniformRandomCenters
(
random
,
numCenter
,
dimension
,
range
);
double
[]
point
=
new
double
[
dimension
];
int
nextCentroid
=
0
;
while
(
true
)
{
// generate a point for the current centroid
double
[]
centroid
=
means
[
nextCentroid
];
for
(
int
d
=
0
;
d
<
dimension
;
d
++)
{
point
[
d
]
=
(
random
.
nextGaussian
()
*
absoluteStdDev
)
+
centroid
[
d
];
}
nextCentroid
=
(
nextCentroid
+
1
)
%
numCenter
;
String
pointString
=
generatePointString
(
point
);
outRecord
.
setString
(
0
,
pointString
);
emit
(
outRecord
);
}
}
private
double
[][]
uniformRandomCenters
(
Random
rnd
,
int
num
,
int
dimensionality
,
double
range
)
{
final
double
halfRange
=
range
/
2
;
final
double
[][]
points
=
new
double
[
num
][
dimensionality
];
for
(
int
i
=
0
;
i
<
num
;
i
++)
{
for
(
int
dim
=
0
;
dim
<
dimensionality
;
dim
++)
{
points
[
i
][
dim
]
=
(
rnd
.
nextDouble
()
*
range
)
-
halfRange
;
}
}
return
points
;
}
private
String
generatePointString
(
double
[]
point
){
buffer
.
setLength
(
0
);
for
(
int
j
=
0
;
j
<
dimension
;
j
++)
{
buffer
.
append
(
point
[
j
]);
if
(
j
<
dimension
-
1
)
{
buffer
.
append
(
" "
);
}
}
return
buffer
.
toString
();
}
}
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/examples/iterative/kmeans/KMeansTask.java
0 → 100644
浏览文件 @
b5dfc948
/***********************************************************************************************************************
*
* Copyright (C) 2010-2014 by the Stratosphere project (http://stratosphere.eu)
*
* Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
* an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
* specific language governing permissions and limitations under the License.
*
**********************************************************************************************************************/
package
eu.stratosphere.streaming.examples.iterative.kmeans
;
import
eu.stratosphere.api.java.tuple.Tuple1
;
import
eu.stratosphere.streaming.api.invokable.UserTaskInvokable
;
import
eu.stratosphere.streaming.api.streamrecord.StreamRecord
;
public
class
KMeansTask
extends
UserTaskInvokable
{
private
static
final
long
serialVersionUID
=
1L
;
private
StreamRecord
outRecord
=
new
StreamRecord
(
new
Tuple1
<
String
>());
private
double
[]
point
=
null
;
public
KMeansTask
(
int
dimension
){
point
=
new
double
[
dimension
];
}
@Override
public
void
invoke
(
StreamRecord
record
)
throws
Exception
{
// TODO Auto-generated method stub
String
[]
pointStr
=
record
.
getString
(
0
,
0
).
split
(
" "
);
for
(
int
i
=
0
;
i
<
pointStr
.
length
;
++
i
){
point
[
i
]=
Double
.
valueOf
(
pointStr
[
i
]);
}
outRecord
.
setString
(
0
,
record
.
getString
(
0
,
0
));
emit
(
outRecord
);
}
}
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/examples/iterative/pagerank/Graph.java
0 → 100644
浏览文件 @
b5dfc948
/***********************************************************************************************************************
*
* Copyright (C) 2010-2014 by the Stratosphere project (http://stratosphere.eu)
*
* Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
* an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
* specific language governing permissions and limitations under the License.
*
**********************************************************************************************************************/
package
eu.stratosphere.streaming.examples.iterative.pagerank
;
import
java.util.HashMap
;
import
java.util.HashSet
;
import
java.util.Map
;
import
java.util.Set
;
public
class
Graph
{
public
Map
<
Integer
,
Set
<
Integer
>>
_vertices
=
null
;
public
Graph
()
{
_vertices
=
new
HashMap
<
Integer
,
Set
<
Integer
>>();
}
public
void
insertDirectedEdge
(
int
sourceNode
,
int
targetNode
)
{
if
(!
_vertices
.
containsKey
(
sourceNode
))
{
_vertices
.
put
(
sourceNode
,
new
HashSet
<
Integer
>());
}
_vertices
.
get
(
sourceNode
).
add
(
targetNode
);
}
public
void
insertUndirectedEdge
(
int
sourceNode
,
int
targetNode
){
if
(!
_vertices
.
containsKey
(
sourceNode
)){
_vertices
.
put
(
sourceNode
,
new
HashSet
<
Integer
>());
}
if
(!
_vertices
.
containsKey
(
targetNode
)){
_vertices
.
put
(
targetNode
,
new
HashSet
<
Integer
>());
}
_vertices
.
get
(
sourceNode
).
add
(
targetNode
);
_vertices
.
get
(
targetNode
).
add
(
sourceNode
);
}
}
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/examples/iterative/pagerank/PageRankLocal.java
0 → 100644
浏览文件 @
b5dfc948
/***********************************************************************************************************************
*
* Copyright (C) 2010-2014 by the Stratosphere project (http://stratosphere.eu)
*
* Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
* an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
* specific language governing permissions and limitations under the License.
*
**********************************************************************************************************************/
package
eu.stratosphere.streaming.examples.iterative.pagerank
;
import
org.apache.log4j.Level
;
import
eu.stratosphere.nephele.jobgraph.JobGraph
;
import
eu.stratosphere.streaming.api.JobGraphBuilder
;
import
eu.stratosphere.streaming.faulttolerance.FaultToleranceType
;
import
eu.stratosphere.streaming.util.ClusterUtil
;
import
eu.stratosphere.streaming.util.LogUtils
;
public
class
PageRankLocal
{
public
static
JobGraph
getJobGraph
()
{
JobGraphBuilder
graphBuilder
=
new
JobGraphBuilder
(
"testGraph"
,
FaultToleranceType
.
NONE
);
graphBuilder
.
setSource
(
"Source"
,
new
PageRankSource
());
graphBuilder
.
setTask
(
"Task"
,
new
PageRankTask
(),
1
,
1
);
graphBuilder
.
setSink
(
"Sink"
,
new
PageRankSink
());
graphBuilder
.
fieldsConnect
(
"Source"
,
"Task"
,
0
);
graphBuilder
.
shuffleConnect
(
"Task"
,
"Sink"
);
return
graphBuilder
.
getJobGraph
();
}
public
static
void
main
(
String
[]
args
)
{
LogUtils
.
initializeDefaultConsoleLogger
(
Level
.
DEBUG
,
Level
.
INFO
);
ClusterUtil
.
runOnMiniCluster
(
getJobGraph
());
}
}
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/examples/iterative/pagerank/PageRankSink.java
0 → 100644
浏览文件 @
b5dfc948
/***********************************************************************************************************************
*
* Copyright (C) 2010-2014 by the Stratosphere project (http://stratosphere.eu)
*
* Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
* an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
* specific language governing permissions and limitations under the License.
*
**********************************************************************************************************************/
package
eu.stratosphere.streaming.examples.iterative.pagerank
;
import
eu.stratosphere.streaming.api.invokable.UserSinkInvokable
;
import
eu.stratosphere.streaming.api.streamrecord.StreamRecord
;
public
class
PageRankSink
extends
UserSinkInvokable
{
private
static
final
long
serialVersionUID
=
1L
;
@Override
public
void
invoke
(
StreamRecord
record
)
throws
Exception
{
// TODO Auto-generated method stub
System
.
out
.
println
(
"received record..."
);
int
tupleNum
=
record
.
getNumOfTuples
();
System
.
out
.
println
(
"============================================"
);
for
(
int
i
=
0
;
i
<
tupleNum
;
++
i
)
{
System
.
out
.
println
(
"name="
+
record
.
getField
(
i
,
0
)
+
", grade="
+
record
.
getField
(
i
,
1
)
+
", salary="
+
record
.
getField
(
i
,
2
));
}
System
.
out
.
println
(
"============================================"
);
}
}
\ No newline at end of file
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/examples/iterative/pagerank/PageRankSource.java
0 → 100644
浏览文件 @
b5dfc948
/***********************************************************************************************************************
*
* Copyright (C) 2010-2014 by the Stratosphere project (http://stratosphere.eu)
*
* Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
* an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
* specific language governing permissions and limitations under the License.
*
**********************************************************************************************************************/
package
eu.stratosphere.streaming.examples.iterative.pagerank
;
import
java.io.BufferedReader
;
import
java.io.FileReader
;
import
eu.stratosphere.api.java.tuple.Tuple2
;
import
eu.stratosphere.streaming.api.invokable.UserSourceInvokable
;
import
eu.stratosphere.streaming.api.streamrecord.StreamRecord
;
public
class
PageRankSource
extends
UserSourceInvokable
{
private
static
final
long
serialVersionUID
=
1L
;
private
BufferedReader
br
=
null
;
private
StreamRecord
outRecord
=
new
StreamRecord
(
new
Tuple2
<
Integer
,
Integer
>());
@Override
public
void
invoke
()
throws
Exception
{
// TODO Auto-generated method stub
br
=
new
BufferedReader
(
new
FileReader
(
"src/test/resources/testdata/ASTopology.data"
));
while
(
true
)
{
String
line
=
br
.
readLine
();
if
(
line
==
null
)
{
break
;
}
if
(
line
!=
""
)
{
String
[]
link
=
line
.
split
(
":"
);
outRecord
.
setInteger
(
0
,
Integer
.
valueOf
(
link
[
0
]));
outRecord
.
setInteger
(
0
,
Integer
.
valueOf
(
link
[
1
]));
emit
(
outRecord
);
performanceCounter
.
count
();
}
line
=
br
.
readLine
();
}
}
}
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/examples/iterative/pagerank/Page
rankLocal
.java
→
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/examples/iterative/pagerank/Page
RankTask
.java
浏览文件 @
b5dfc948
...
...
@@ -15,11 +15,25 @@
package
eu.stratosphere.streaming.examples.iterative.pagerank
;
public
class
PagerankLocal
{
import
eu.stratosphere.api.java.tuple.Tuple1
;
import
eu.stratosphere.streaming.api.invokable.UserTaskInvokable
;
import
eu.stratosphere.streaming.api.streamrecord.StreamRecord
;
public
static
void
main
(
String
[]
args
)
{
// TODO Auto-generated method stub
public
class
PageRankTask
extends
UserTaskInvokable
{
private
static
final
long
serialVersionUID
=
1L
;
private
StreamRecord
outRecord
=
new
StreamRecord
(
new
Tuple1
<
String
>());
private
Graph
linkGraph
=
new
Graph
();
@Override
public
void
invoke
(
StreamRecord
record
)
throws
Exception
{
// TODO Auto-generated method stub
Integer
sourceNode
=
record
.
getInteger
(
0
,
0
);
Integer
targetNode
=
record
.
getInteger
(
0
,
1
);
// set the input graph.
linkGraph
.
insertDirectedEdge
(
sourceNode
,
targetNode
);
//outRecord.setString(0, line);
}
}
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/examples/iterative/sssp/Graph.java
0 → 100644
浏览文件 @
b5dfc948
/***********************************************************************************************************************
*
* Copyright (C) 2010-2014 by the Stratosphere project (http://stratosphere.eu)
*
* Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
* an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
* specific language governing permissions and limitations under the License.
*
**********************************************************************************************************************/
package
eu.stratosphere.streaming.examples.iterative.sssp
;
import
java.util.HashMap
;
import
java.util.HashSet
;
import
java.util.Map
;
import
java.util.Set
;
public
class
Graph
{
public
Map
<
Integer
,
Set
<
Integer
>>
_vertices
=
null
;
public
Graph
()
{
_vertices
=
new
HashMap
<
Integer
,
Set
<
Integer
>>();
}
public
void
insertDirectedEdge
(
int
sourceNode
,
int
targetNode
)
{
if
(!
_vertices
.
containsKey
(
sourceNode
))
{
_vertices
.
put
(
sourceNode
,
new
HashSet
<
Integer
>());
}
_vertices
.
get
(
sourceNode
).
add
(
targetNode
);
}
public
void
insertUndirectedEdge
(
int
sourceNode
,
int
targetNode
){
if
(!
_vertices
.
containsKey
(
sourceNode
)){
_vertices
.
put
(
sourceNode
,
new
HashSet
<
Integer
>());
}
if
(!
_vertices
.
containsKey
(
targetNode
)){
_vertices
.
put
(
targetNode
,
new
HashSet
<
Integer
>());
}
_vertices
.
get
(
sourceNode
).
add
(
targetNode
);
_vertices
.
get
(
targetNode
).
add
(
sourceNode
);
}
}
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/examples/iterative/sssp/SSSPLocal.java
浏览文件 @
b5dfc948
...
...
@@ -15,11 +15,32 @@
package
eu.stratosphere.streaming.examples.iterative.sssp
;
import
org.apache.log4j.Level
;
import
eu.stratosphere.nephele.jobgraph.JobGraph
;
import
eu.stratosphere.streaming.api.JobGraphBuilder
;
import
eu.stratosphere.streaming.faulttolerance.FaultToleranceType
;
import
eu.stratosphere.streaming.util.ClusterUtil
;
import
eu.stratosphere.streaming.util.LogUtils
;
public
class
SSSPLocal
{
public
static
JobGraph
getJobGraph
()
{
JobGraphBuilder
graphBuilder
=
new
JobGraphBuilder
(
"testGraph"
,
FaultToleranceType
.
NONE
);
graphBuilder
.
setSource
(
"Source"
,
new
SSSPSource
());
graphBuilder
.
setTask
(
"Task"
,
new
SSSPTask
(),
1
,
1
);
graphBuilder
.
setSink
(
"Sink"
,
new
SSSPSink
());
public
static
void
main
(
String
[]
args
)
{
// TODO Auto-generated method stub
graphBuilder
.
fieldsConnect
(
"Source"
,
"Task"
,
0
);
graphBuilder
.
shuffleConnect
(
"Task"
,
"Sink"
);
return
graphBuilder
.
getJobGraph
();
}
public
static
void
main
(
String
[]
args
)
{
LogUtils
.
initializeDefaultConsoleLogger
(
Level
.
DEBUG
,
Level
.
INFO
);
ClusterUtil
.
runOnMiniCluster
(
getJobGraph
());
}
}
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/examples/iterative/sssp/SSSPSink.java
0 → 100644
浏览文件 @
b5dfc948
/***********************************************************************************************************************
*
* Copyright (C) 2010-2014 by the Stratosphere project (http://stratosphere.eu)
*
* Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
* an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
* specific language governing permissions and limitations under the License.
*
**********************************************************************************************************************/
package
eu.stratosphere.streaming.examples.iterative.sssp
;
import
eu.stratosphere.streaming.api.invokable.UserSinkInvokable
;
import
eu.stratosphere.streaming.api.streamrecord.StreamRecord
;
public
class
SSSPSink
extends
UserSinkInvokable
{
private
static
final
long
serialVersionUID
=
1L
;
@Override
public
void
invoke
(
StreamRecord
record
)
throws
Exception
{
// TODO Auto-generated method stub
System
.
out
.
println
(
"received record..."
);
int
tupleNum
=
record
.
getNumOfTuples
();
System
.
out
.
println
(
"============================================"
);
for
(
int
i
=
0
;
i
<
tupleNum
;
++
i
)
{
System
.
out
.
println
(
"name="
+
record
.
getField
(
i
,
0
)
+
", grade="
+
record
.
getField
(
i
,
1
)
+
", salary="
+
record
.
getField
(
i
,
2
));
}
System
.
out
.
println
(
"============================================"
);
}
}
\ No newline at end of file
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/examples/iterative/sssp/SSSPSource.java
0 → 100644
浏览文件 @
b5dfc948
/***********************************************************************************************************************
*
* Copyright (C) 2010-2014 by the Stratosphere project (http://stratosphere.eu)
*
* Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
* an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
* specific language governing permissions and limitations under the License.
*
**********************************************************************************************************************/
package
eu.stratosphere.streaming.examples.iterative.sssp
;
import
java.io.BufferedReader
;
import
java.io.FileReader
;
import
eu.stratosphere.api.java.tuple.Tuple2
;
import
eu.stratosphere.streaming.api.invokable.UserSourceInvokable
;
import
eu.stratosphere.streaming.api.streamrecord.StreamRecord
;
public
class
SSSPSource
extends
UserSourceInvokable
{
private
static
final
long
serialVersionUID
=
1L
;
private
BufferedReader
br
=
null
;
private
StreamRecord
outRecord
=
new
StreamRecord
(
new
Tuple2
<
Integer
,
Integer
>());
@Override
public
void
invoke
()
throws
Exception
{
// TODO Auto-generated method stub
br
=
new
BufferedReader
(
new
FileReader
(
"src/test/resources/testdata/ASTopology.data"
));
while
(
true
)
{
String
line
=
br
.
readLine
();
if
(
line
==
null
)
{
break
;
}
if
(
line
!=
""
)
{
String
[]
link
=
line
.
split
(
":"
);
outRecord
.
setInteger
(
0
,
Integer
.
valueOf
(
link
[
0
]));
outRecord
.
setInteger
(
0
,
Integer
.
valueOf
(
link
[
1
]));
emit
(
outRecord
);
performanceCounter
.
count
();
}
line
=
br
.
readLine
();
}
}
}
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/examples/iterative/
collaborativefilter/CF
Task.java
→
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/examples/iterative/
sssp/SSSP
Task.java
浏览文件 @
b5dfc948
...
...
@@ -13,19 +13,27 @@
*
**********************************************************************************************************************/
package
eu.stratosphere.streaming.examples.iterative.
collaborativefilter
;
package
eu.stratosphere.streaming.examples.iterative.
sssp
;
import
eu.stratosphere.api.java.tuple.Tuple1
;
import
eu.stratosphere.streaming.api.invokable.UserTaskInvokable
;
import
eu.stratosphere.streaming.api.streamrecord.StreamRecord
;
public
class
CF
Task
extends
UserTaskInvokable
{
public
class
SSSP
Task
extends
UserTaskInvokable
{
private
static
final
long
serialVersionUID
=
1L
;
private
StreamRecord
outRecord
=
new
StreamRecord
(
new
Tuple1
<
String
>());
private
Graph
linkGraph
=
new
Graph
();
@Override
public
void
invoke
(
StreamRecord
record
)
throws
Exception
{
// TODO Auto-generated method stub
Integer
sourceNode
=
record
.
getInteger
(
0
,
0
);
Integer
targetNode
=
record
.
getInteger
(
0
,
1
);
// set the input graph.
linkGraph
.
insertDirectedEdge
(
sourceNode
,
targetNode
);
//outRecord.setString(0, line);
}
}
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/examples/join/JoinLocal.java
浏览文件 @
b5dfc948
...
...
@@ -15,16 +15,12 @@
package
eu.stratosphere.streaming.examples.join
;
import
java.net.InetSocketAddress
;
import
org.apache.log4j.Level
;
import
eu.stratosphere.client.minicluster.NepheleMiniCluster
;
import
eu.stratosphere.client.program.Client
;
import
eu.stratosphere.configuration.Configuration
;
import
eu.stratosphere.nephele.jobgraph.JobGraph
;
import
eu.stratosphere.streaming.api.JobGraphBuilder
;
import
eu.stratosphere.streaming.faulttolerance.FaultToleranceType
;
import
eu.stratosphere.streaming.util.ClusterUtil
;
import
eu.stratosphere.streaming.util.LogUtils
;
public
class
JoinLocal
{
...
...
@@ -46,41 +42,7 @@ public class JoinLocal {
public
static
void
main
(
String
[]
args
)
{
LogUtils
.
initializeDefaultConsoleLogger
(
Level
.
DEBUG
,
Level
.
INFO
);
try
{
JobGraph
jG
=
getJobGraph
();
Configuration
configuration
=
jG
.
getJobConfiguration
();
if
(
args
.
length
==
0
)
{
args
=
new
String
[]
{
"local"
};
}
if
(
args
[
0
].
equals
(
"local"
))
{
System
.
out
.
println
(
"Running in Local mode"
);
NepheleMiniCluster
exec
=
new
NepheleMiniCluster
();
exec
.
start
();
Client
client
=
new
Client
(
new
InetSocketAddress
(
"localhost"
,
6498
),
configuration
);
client
.
run
(
jG
,
true
);
exec
.
stop
();
}
else
if
(
args
[
0
].
equals
(
"cluster"
))
{
System
.
out
.
println
(
"Running in Cluster2 mode"
);
Client
client
=
new
Client
(
new
InetSocketAddress
(
"hadoop02.ilab.sztaki.hu"
,
6123
),
configuration
);
client
.
run
(
jG
,
true
);
}
}
catch
(
Exception
e
)
{
System
.
out
.
println
(
e
);
}
ClusterUtil
.
runOnMiniCluster
(
getJobGraph
());
}
}
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/examples/window/join/WindowJoinLocal.java
浏览文件 @
b5dfc948
...
...
@@ -15,16 +15,12 @@
package
eu.stratosphere.streaming.examples.window.join
;
import
java.net.InetSocketAddress
;
import
org.apache.log4j.Level
;
import
eu.stratosphere.client.minicluster.NepheleMiniCluster
;
import
eu.stratosphere.client.program.Client
;
import
eu.stratosphere.configuration.Configuration
;
import
eu.stratosphere.nephele.jobgraph.JobGraph
;
import
eu.stratosphere.streaming.api.JobGraphBuilder
;
import
eu.stratosphere.streaming.faulttolerance.FaultToleranceType
;
import
eu.stratosphere.streaming.util.ClusterUtil
;
import
eu.stratosphere.streaming.util.LogUtils
;
public
class
WindowJoinLocal
{
...
...
@@ -46,41 +42,7 @@ public class WindowJoinLocal {
public
static
void
main
(
String
[]
args
)
{
LogUtils
.
initializeDefaultConsoleLogger
(
Level
.
DEBUG
,
Level
.
INFO
);
try
{
JobGraph
jG
=
getJobGraph
();
Configuration
configuration
=
jG
.
getJobConfiguration
();
if
(
args
.
length
==
0
)
{
args
=
new
String
[]
{
"local"
};
}
if
(
args
[
0
].
equals
(
"local"
))
{
System
.
out
.
println
(
"Running in Local mode"
);
NepheleMiniCluster
exec
=
new
NepheleMiniCluster
();
exec
.
start
();
Client
client
=
new
Client
(
new
InetSocketAddress
(
"localhost"
,
6498
),
configuration
);
client
.
run
(
jG
,
true
);
exec
.
stop
();
}
else
if
(
args
[
0
].
equals
(
"cluster"
))
{
System
.
out
.
println
(
"Running in Cluster2 mode"
);
Client
client
=
new
Client
(
new
InetSocketAddress
(
"hadoop02.ilab.sztaki.hu"
,
6123
),
configuration
);
client
.
run
(
jG
,
true
);
}
}
catch
(
Exception
e
)
{
System
.
out
.
println
(
e
);
}
ClusterUtil
.
runOnMiniCluster
(
getJobGraph
());
}
}
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/examples/window/sum/WindowSumLocal.java
浏览文件 @
b5dfc948
...
...
@@ -15,15 +15,11 @@
package
eu.stratosphere.streaming.examples.window.sum
;
import
java.net.InetSocketAddress
;
import
org.apache.log4j.Level
;
import
eu.stratosphere.client.minicluster.NepheleMiniCluster
;
import
eu.stratosphere.client.program.Client
;
import
eu.stratosphere.configuration.Configuration
;
import
eu.stratosphere.nephele.jobgraph.JobGraph
;
import
eu.stratosphere.streaming.api.JobGraphBuilder
;
import
eu.stratosphere.streaming.util.ClusterUtil
;
import
eu.stratosphere.streaming.util.LogUtils
;
public
class
WindowSumLocal
{
...
...
@@ -45,41 +41,7 @@ public class WindowSumLocal {
public
static
void
main
(
String
[]
args
)
{
LogUtils
.
initializeDefaultConsoleLogger
(
Level
.
DEBUG
,
Level
.
INFO
);
try
{
JobGraph
jG
=
getJobGraph
();
Configuration
configuration
=
jG
.
getJobConfiguration
();
if
(
args
.
length
==
0
)
{
args
=
new
String
[]
{
"local"
};
}
if
(
args
[
0
].
equals
(
"local"
))
{
System
.
out
.
println
(
"Running in Local mode"
);
NepheleMiniCluster
exec
=
new
NepheleMiniCluster
();
exec
.
start
();
Client
client
=
new
Client
(
new
InetSocketAddress
(
"localhost"
,
6498
),
configuration
);
client
.
run
(
jG
,
true
);
exec
.
stop
();
}
else
if
(
args
[
0
].
equals
(
"cluster"
))
{
System
.
out
.
println
(
"Running in Cluster2 mode"
);
Client
client
=
new
Client
(
new
InetSocketAddress
(
"hadoop02.ilab.sztaki.hu"
,
6123
),
configuration
);
client
.
run
(
jG
,
true
);
}
}
catch
(
Exception
e
)
{
System
.
out
.
println
(
e
);
}
ClusterUtil
.
runOnMiniCluster
(
getJobGraph
());
}
}
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/examples/window/wordcount/WindowWordCountLocal.java
浏览文件 @
b5dfc948
...
...
@@ -15,16 +15,12 @@
package
eu.stratosphere.streaming.examples.window.wordcount
;
import
java.net.InetSocketAddress
;
import
org.apache.log4j.Level
;
import
eu.stratosphere.client.minicluster.NepheleMiniCluster
;
import
eu.stratosphere.client.program.Client
;
import
eu.stratosphere.configuration.Configuration
;
import
eu.stratosphere.nephele.jobgraph.JobGraph
;
import
eu.stratosphere.streaming.api.JobGraphBuilder
;
import
eu.stratosphere.streaming.faulttolerance.FaultToleranceType
;
import
eu.stratosphere.streaming.util.ClusterUtil
;
import
eu.stratosphere.streaming.util.LogUtils
;
public
class
WindowWordCountLocal
{
...
...
@@ -46,41 +42,7 @@ public class WindowWordCountLocal {
public
static
void
main
(
String
[]
args
)
{
LogUtils
.
initializeDefaultConsoleLogger
(
Level
.
DEBUG
,
Level
.
INFO
);
try
{
JobGraph
jG
=
getJobGraph
();
Configuration
configuration
=
jG
.
getJobConfiguration
();
if
(
args
.
length
==
0
)
{
args
=
new
String
[]
{
"local"
};
}
if
(
args
[
0
].
equals
(
"local"
))
{
System
.
out
.
println
(
"Running in Local mode"
);
NepheleMiniCluster
exec
=
new
NepheleMiniCluster
();
exec
.
start
();
Client
client
=
new
Client
(
new
InetSocketAddress
(
"localhost"
,
6498
),
configuration
);
client
.
run
(
jG
,
true
);
exec
.
stop
();
}
else
if
(
args
[
0
].
equals
(
"cluster"
))
{
System
.
out
.
println
(
"Running in Cluster2 mode"
);
Client
client
=
new
Client
(
new
InetSocketAddress
(
"hadoop02.ilab.sztaki.hu"
,
6123
),
configuration
);
client
.
run
(
jG
,
true
);
}
}
catch
(
Exception
e
)
{
System
.
out
.
println
(
e
);
}
ClusterUtil
.
runOnMiniCluster
(
getJobGraph
());
}
}
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/examples/wordcount/WordCountLocal.java
浏览文件 @
b5dfc948
...
...
@@ -41,17 +41,6 @@ public class WordCountLocal {
public
static
void
main
(
String
[]
args
)
{
LogUtils
.
initializeDefaultConsoleLogger
(
Level
.
DEBUG
,
Level
.
INFO
);
if
(
args
.
length
==
0
)
{
args
=
new
String
[]
{
"local"
};
}
if
(
args
[
0
].
equals
(
"local"
))
{
ClusterUtil
.
runOnMiniCluster
(
getJobGraph
());
}
else
if
(
args
[
0
].
equals
(
"cluster"
))
{
ClusterUtil
.
runOnLocalCluster
(
getJobGraph
(),
"hadoop02.ilab.sztaki.hu"
,
6123
);
}
ClusterUtil
.
runOnMiniCluster
(
getJobGraph
());
}
}
flink-addons/flink-streaming/src/main/java/helloworld.java
已删除
100644 → 0
浏览文件 @
c92727b8
public
class
helloworld
{
public
static
void
main
(
String
[]
args
)
{
// TODO Auto-generated method stub
}
}
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录