Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
doujutun3207
flink
提交
900c8d8e
F
flink
项目概览
doujutun3207
/
flink
与 Fork 源项目一致
从无法访问的项目Fork
通知
24
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
F
flink
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
900c8d8e
编写于
7月 14, 2014
作者:
G
gyfora
提交者:
Stephan Ewen
8月 18, 2014
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
[streaming] incremental machine learning skeleton added
上级
08d2ac62
变更
2
隐藏空白更改
内联
并排
Showing
2 changed file
with
136 addition
and
0 deletion
+136
-0
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/api/streamrecord/StreamRecord.java
...stratosphere/streaming/api/streamrecord/StreamRecord.java
+11
-0
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/examples/ml/IncrementalLearning.java
...ratosphere/streaming/examples/ml/IncrementalLearning.java
+125
-0
未找到文件。
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/api/streamrecord/StreamRecord.java
浏览文件 @
900c8d8e
...
...
@@ -72,6 +72,7 @@ public class StreamRecord implements IOReadableWritable, Serializable {
private
UID
uid
=
new
UID
();
private
int
numOfFields
;
private
int
numOfTuples
;
private
int
batchSize
;
private
static
final
Class
<?>[]
CLASSES
=
new
Class
<?>[]
{
Tuple1
.
class
,
Tuple2
.
class
,
Tuple3
.
class
,
Tuple4
.
class
,
Tuple5
.
class
,
Tuple6
.
class
,
Tuple7
.
class
,
Tuple8
.
class
,
Tuple9
.
class
,
Tuple10
.
class
,
Tuple11
.
class
,
...
...
@@ -95,6 +96,7 @@ public class StreamRecord implements IOReadableWritable, Serializable {
public
StreamRecord
(
int
numOfFields
,
int
batchSize
)
{
this
.
numOfFields
=
numOfFields
;
this
.
numOfTuples
=
0
;
this
.
batchSize
=
batchSize
;
tupleBatch
=
new
ArrayList
<
Tuple
>(
batchSize
);
}
...
...
@@ -111,6 +113,7 @@ public class StreamRecord implements IOReadableWritable, Serializable {
public
StreamRecord
(
Tuple
tuple
,
int
batchSize
)
{
numOfFields
=
tuple
.
getArity
();
numOfTuples
=
1
;
this
.
batchSize
=
batchSize
;
tupleBatch
=
new
ArrayList
<
Tuple
>(
batchSize
);
tupleBatch
.
add
(
tuple
);
...
...
@@ -159,6 +162,14 @@ public class StreamRecord implements IOReadableWritable, Serializable {
uid
=
new
UID
(
channelID
);
return
this
;
}
public
void
InitRecords
(){
tupleBatch
.
clear
();
for
(
int
i
=
0
;
i
<
batchSize
;
i
++){
tupleBatch
.
add
(
null
);
}
numOfTuples
=
batchSize
;
}
/**
* Returns the value of a field in the given position of the first tuple in
...
...
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/examples/ml/IncrementalLearning.java
0 → 100755
浏览文件 @
900c8d8e
package
eu.stratosphere.streaming.examples.ml
;
import
java.net.InetSocketAddress
;
import
org.apache.log4j.Level
;
import
eu.stratosphere.api.java.tuple.Tuple
;
import
eu.stratosphere.api.java.tuple.Tuple1
;
import
eu.stratosphere.client.minicluster.NepheleMiniCluster
;
import
eu.stratosphere.client.program.Client
;
import
eu.stratosphere.configuration.Configuration
;
import
eu.stratosphere.nephele.jobgraph.JobGraph
;
import
eu.stratosphere.streaming.api.JobGraphBuilder
;
import
eu.stratosphere.streaming.api.invokable.UserSinkInvokable
;
import
eu.stratosphere.streaming.api.invokable.UserSourceInvokable
;
import
eu.stratosphere.streaming.api.invokable.UserTaskInvokable
;
import
eu.stratosphere.streaming.api.streamrecord.StreamRecord
;
import
eu.stratosphere.streaming.util.LogUtils
;
public
class
IncrementalLearning
{
public
static
class
NewDataSource
extends
UserSourceInvokable
{
StreamRecord
record
=
new
StreamRecord
(
new
Tuple1
<
Integer
>(
1
));
@Override
public
void
invoke
()
throws
Exception
{
while
(
true
)
{
// pull new record from data source
record
.
setTuple
(
getNewData
());
emit
(
record
);
}
}
private
Tuple
getNewData
()
throws
InterruptedException
{
Thread
.
sleep
(
1000
);
return
new
Tuple1
<
Integer
>(
1
);
}
}
public
static
class
OldDataSource
extends
UserSourceInvokable
{
private
final
int
BATCH_SIZE
=
1000
;
StreamRecord
record
=
new
StreamRecord
(
2
,
BATCH_SIZE
);
@Override
public
void
invoke
()
throws
Exception
{
record
.
InitRecords
();
while
(
true
)
{
for
(
int
i
=
0
;
i
<
BATCH_SIZE
;
i
++)
{
record
.
setTuple
(
getOldData
());
}
emit
(
record
);
}
}
private
Tuple
getOldData
()
throws
InterruptedException
{
Thread
.
sleep
(
1000
);
return
new
Tuple1
<
Integer
>(
1
);
}
}
public
static
class
WindowPredictor
extends
UserTaskInvokable
{
@Override
public
void
invoke
(
StreamRecord
record
)
throws
Exception
{
//build predictor on record
}
}
public
static
class
Sink
extends
UserSinkInvokable
{
@Override
public
void
invoke
(
StreamRecord
record
)
throws
Exception
{
// do nothing
}
}
private
static
JobGraph
getJobGraph
()
throws
Exception
{
JobGraphBuilder
graphBuilder
=
new
JobGraphBuilder
(
"IncrementalLearning"
);
return
graphBuilder
.
getJobGraph
();
}
public
static
void
main
(
String
[]
args
)
{
// set logging parameters for local run
LogUtils
.
initializeDefaultConsoleLogger
(
Level
.
ERROR
,
Level
.
INFO
);
try
{
// generate JobGraph
JobGraph
jG
=
getJobGraph
();
Configuration
configuration
=
jG
.
getJobConfiguration
();
if
(
args
.
length
==
0
||
args
[
0
].
equals
(
"local"
))
{
System
.
out
.
println
(
"Running in Local mode"
);
// start local cluster and submit JobGraph
NepheleMiniCluster
exec
=
new
NepheleMiniCluster
();
exec
.
start
();
Client
client
=
new
Client
(
new
InetSocketAddress
(
"localhost"
,
6498
),
configuration
);
client
.
run
(
jG
,
true
);
exec
.
stop
();
}
else
if
(
args
[
0
].
equals
(
"cluster"
))
{
System
.
out
.
println
(
"Running in Cluster mode"
);
// submit JobGraph to the running cluster
Client
client
=
new
Client
(
new
InetSocketAddress
(
"dell150"
,
6123
),
configuration
);
client
.
run
(
jG
,
true
);
}
}
catch
(
Exception
e
)
{
System
.
out
.
println
(
e
);
}
}
}
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录