Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
doujutun3207
flink
提交
a3b1dfa0
F
flink
项目概览
doujutun3207
/
flink
与 Fork 源项目一致
从无法访问的项目Fork
通知
24
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
F
flink
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
a3b1dfa0
编写于
7月 14, 2014
作者:
M
Márton Balassi
提交者:
Stephan Ewen
8月 18, 2014
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
[streaming] Project version set to 0.1
上级
a1116c3f
变更
14
展开全部
隐藏空白更改
内联
并排
Showing
14 changed file
with
1520 addition
and
1585 deletion
+1520
-1585
flink-addons/flink-streaming/pom.xml
flink-addons/flink-streaming/pom.xml
+8
-7
flink-addons/flink-streaming/src/main/java/eu/stratosphere/api/datastream/DataStream.java
.../main/java/eu/stratosphere/api/datastream/DataStream.java
+0
-29
flink-addons/flink-streaming/src/main/java/eu/stratosphere/api/datastream/SingleStreamInputOperator.java
...tratosphere/api/datastream/SingleStreamInputOperator.java
+0
-12
flink-addons/flink-streaming/src/main/java/eu/stratosphere/api/datastream/StreamExecutionEnvironment.java
...ratosphere/api/datastream/StreamExecutionEnvironment.java
+0
-21
flink-addons/flink-streaming/src/main/java/eu/stratosphere/api/datastream/StreamOperator.java
...n/java/eu/stratosphere/api/datastream/StreamOperator.java
+0
-10
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/api/StreamCollector.java
...n/java/eu/stratosphere/streaming/api/StreamCollector.java
+0
-41
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/api/streamrecord/StreamRecord.java
...stratosphere/streaming/api/streamrecord/StreamRecord.java
+1067
-35
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/api/streamrecord/StreamRecordGeneric.java
...phere/streaming/api/streamrecord/StreamRecordGeneric.java
+0
-1351
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/examples/wordcount/WordCountCounter.java
...sphere/streaming/examples/wordcount/WordCountCounter.java
+3
-5
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/examples/wordcount/WordCountSourceSplitter.java
...streaming/examples/wordcount/WordCountSourceSplitter.java
+1
-1
flink-addons/flink-streaming/src/test/java/eu/stratosphere/streaming/api/StreamCollectorTest.java
...va/eu/stratosphere/streaming/api/StreamCollectorTest.java
+0
-31
flink-addons/flink-streaming/src/test/java/eu/stratosphere/streaming/api/streamrecord/MyGeneric.java
...eu/stratosphere/streaming/api/streamrecord/MyGeneric.java
+0
-9
flink-addons/flink-streaming/src/test/java/eu/stratosphere/streaming/api/streamrecord/MyGeneric2.java
...u/stratosphere/streaming/api/streamrecord/MyGeneric2.java
+0
-11
flink-addons/flink-streaming/src/test/java/eu/stratosphere/streaming/api/streamrecord/StreamRecordTest.java
...tosphere/streaming/api/streamrecord/StreamRecordTest.java
+441
-22
未找到文件。
flink-addons/flink-streaming/pom.xml
浏览文件 @
a3b1dfa0
...
...
@@ -5,13 +5,14 @@
<modelVersion>
4.0.0
</modelVersion>
<groupId>
eu.stratosphere
</groupId>
<version>
0.
5
</version>
<version>
0.
1
</version>
<artifactId>
stratosphere-streaming
</artifactId>
<name>
stratosphere-streaming
</name>
<packaging>
jar
</packaging>
<properties>
<stratosphere.version>
0.5
</stratosphere.version>
<project.build.sourceEncoding>
UTF-8
</project.build.sourceEncoding>
<project.reporting.outputEncoding>
UTF-8
</project.reporting.outputEncoding>
</properties>
...
...
@@ -29,32 +30,32 @@
<dependency>
<groupId>
eu.stratosphere
</groupId>
<artifactId>
stratosphere-core
</artifactId>
<version>
${
project
.version}
</version>
<version>
${
stratosphere
.version}
</version>
</dependency>
<dependency>
<groupId>
eu.stratosphere
</groupId>
<artifactId>
stratosphere-tests
</artifactId>
<version>
${
project
.version}
</version>
<version>
${
stratosphere
.version}
</version>
</dependency>
<dependency>
<groupId>
eu.stratosphere
</groupId>
<artifactId>
stratosphere-compiler
</artifactId>
<version>
${
project
.version}
</version>
<version>
${
stratosphere
.version}
</version>
</dependency>
<dependency>
<groupId>
eu.stratosphere
</groupId>
<artifactId>
stratosphere-runtime
</artifactId>
<version>
${
project
.version}
</version>
<version>
${
stratosphere
.version}
</version>
</dependency>
<dependency>
<groupId>
eu.stratosphere
</groupId>
<artifactId>
stratosphere-clients
</artifactId>
<version>
${
project
.version}
</version>
<version>
${
stratosphere
.version}
</version>
</dependency>
<dependency>
<groupId>
eu.stratosphere
</groupId>
<artifactId>
stratosphere-java
</artifactId>
<version>
${
project
.version}
</version>
<version>
${
stratosphere
.version}
</version>
</dependency>
<dependency>
<groupId>
junit
</groupId>
...
...
flink-addons/flink-streaming/src/main/java/eu/stratosphere/api/datastream/DataStream.java
已删除
100644 → 0
浏览文件 @
a1116c3f
package
eu.stratosphere.api.datastream
;
import
eu.stratosphere.types.TypeInformation
;
public
class
DataStream
<
T
>
{
private
final
StreamExecutionEnvironment
context
;
private
final
TypeInformation
<
T
>
type
;
protected
DataStream
(
StreamExecutionEnvironment
context
,
TypeInformation
<
T
>
type
)
{
if
(
context
==
null
)
{
throw
new
NullPointerException
(
"context is null"
);
}
if
(
type
==
null
)
{
throw
new
NullPointerException
(
"type is null"
);
}
this
.
context
=
context
;
this
.
type
=
type
;
}
public
TypeInformation
<
T
>
getType
()
{
return
this
.
type
;
}
}
\ No newline at end of file
flink-addons/flink-streaming/src/main/java/eu/stratosphere/api/datastream/SingleStreamInputOperator.java
已删除
100644 → 0
浏览文件 @
a1116c3f
package
eu.stratosphere.api.datastream
;
import
eu.stratosphere.types.TypeInformation
;
public
abstract
class
SingleStreamInputOperator
<
IN
,
OUT
,
O
extends
SingleStreamInputOperator
<
IN
,
OUT
,
O
>>
extends
StreamOperator
<
OUT
,
O
>
{
protected
SingleStreamInputOperator
(
StreamExecutionEnvironment
context
,
TypeInformation
<
OUT
>
type
)
{
super
(
context
,
type
);
}
}
flink-addons/flink-streaming/src/main/java/eu/stratosphere/api/datastream/StreamExecutionEnvironment.java
已删除
100644 → 0
浏览文件 @
a1116c3f
package
eu.stratosphere.api.datastream
;
import
eu.stratosphere.streaming.api.JobGraphBuilder
;
import
eu.stratosphere.streaming.faulttolerance.FaultToleranceType
;
import
eu.stratosphere.streaming.util.ClusterUtil
;
public
class
StreamExecutionEnvironment
{
JobGraphBuilder
jobGraphBuilder
;
public
StreamExecutionEnvironment
()
{
jobGraphBuilder
=
new
JobGraphBuilder
(
"jobGraph"
,
FaultToleranceType
.
NONE
);
}
// public static StreamExecutionEnvironment getLocalEnvironment() {
// return new StreamExecutionEnvironment();
// }
public
void
execute
()
{
ClusterUtil
.
runOnMiniCluster
(
jobGraphBuilder
.
getJobGraph
());
}
}
flink-addons/flink-streaming/src/main/java/eu/stratosphere/api/datastream/StreamOperator.java
已删除
100644 → 0
浏览文件 @
a1116c3f
package
eu.stratosphere.api.datastream
;
import
eu.stratosphere.types.TypeInformation
;
public
abstract
class
StreamOperator
<
OUT
,
O
extends
StreamOperator
<
OUT
,
O
>>
extends
DataStream
<
OUT
>
{
protected
StreamOperator
(
StreamExecutionEnvironment
context
,
TypeInformation
<
OUT
>
type
)
{
super
(
context
,
type
);
}
}
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/api/StreamCollector.java
已删除
100755 → 0
浏览文件 @
a1116c3f
package
eu.stratosphere.streaming.api
;
import
eu.stratosphere.api.java.tuple.Tuple
;
import
eu.stratosphere.streaming.api.streamrecord.StreamRecord
;
import
eu.stratosphere.util.Collector
;
public
class
StreamCollector
implements
Collector
<
Tuple
>
{
protected
StreamRecord
streamRecord
;
protected
int
batchSize
;
protected
int
counter
=
0
;
protected
int
channelID
;
public
StreamCollector
(
int
batchSize
,
int
channelID
)
{
this
.
batchSize
=
batchSize
;
this
.
streamRecord
=
new
StreamRecord
(
batchSize
);
this
.
channelID
=
channelID
;
}
@Override
public
void
collect
(
Tuple
tuple
)
{
streamRecord
.
setTuple
(
counter
,
StreamRecord
.
copyTuple
(
tuple
));
counter
++;
if
(
counter
>=
batchSize
)
{
counter
=
0
;
streamRecord
.
setId
(
channelID
);
emit
(
streamRecord
);
}
}
private
void
emit
(
StreamRecord
streamRecord
)
{
System
.
out
.
println
(
streamRecord
);
}
@Override
public
void
close
()
{
// TODO Auto-generated method stub
}
}
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/api/streamrecord/StreamRecord.java
浏览文件 @
a3b1dfa0
此差异已折叠。
点击以展开。
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/api/streamrecord/StreamRecordGeneric.java
已删除
100755 → 0
浏览文件 @
a1116c3f
此差异已折叠。
点击以展开。
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/examples/wordcount/WordCountCounter.java
浏览文件 @
a3b1dfa0
...
...
@@ -15,17 +15,15 @@
package
eu.stratosphere.streaming.examples.wordcount
;
import
java.util.HashMap
;
import
java.util.Map
;
import
eu.stratosphere.api.java.tuple.Tuple2
;
import
eu.stratosphere.streaming.api.invokable.UserTaskInvokable
;
import
eu.stratosphere.streaming.api.streamrecord.StreamRecord
;
import
eu.stratosphere.streaming.state.MutableTableState
;
public
class
WordCountCounter
extends
UserTaskInvokable
{
private
static
final
long
serialVersionUID
=
1L
;
private
M
ap
<
String
,
Integer
>
wordCounts
=
new
HashMap
<
String
,
Integer
>();
private
M
utableTableState
<
String
,
Integer
>
wordCounts
=
new
MutableTableState
<
String
,
Integer
>();
private
String
word
=
""
;
private
Integer
count
=
0
;
...
...
@@ -47,7 +45,7 @@ public class WordCountCounter extends UserTaskInvokable {
outRecord
.
setInteger
(
1
,
count
);
emit
(
outRecord
);
//
performanceCounter.count();
performanceCounter
.
count
();
}
...
...
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/examples/wordcount/WordCountSourceSplitter.java
浏览文件 @
a3b1dfa0
...
...
@@ -48,7 +48,7 @@ public class WordCountSourceSplitter extends UserSourceInvokable {
outRecord
.
setString
(
0
,
word
);
System
.
out
.
println
(
"word="
+
word
);
emit
(
outRecord
);
//
performanceCounter.count();
performanceCounter
.
count
();
}
}
}
...
...
flink-addons/flink-streaming/src/test/java/eu/stratosphere/streaming/api/StreamCollectorTest.java
已删除
100755 → 0
浏览文件 @
a1116c3f
package
eu.stratosphere.streaming.api
;
import
static
org
.
junit
.
Assert
.*;
import
org.junit.Test
;
import
eu.stratosphere.api.java.tuple.Tuple1
;
public
class
StreamCollectorTest
{
@Test
public
void
testStreamCollector
()
{
StreamCollector
collector
=
new
StreamCollector
(
10
,
0
);
assertEquals
(
10
,
collector
.
batchSize
);
}
@Test
public
void
testCollect
()
{
StreamCollector
collector
=
new
StreamCollector
(
2
,
0
);
collector
.
collect
(
new
Tuple1
<
Integer
>(
3
));
collector
.
collect
(
new
Tuple1
<
Integer
>(
4
));
collector
.
collect
(
new
Tuple1
<
Integer
>(
5
));
collector
.
collect
(
new
Tuple1
<
Integer
>(
6
));
}
@Test
public
void
testClose
()
{
}
}
flink-addons/flink-streaming/src/test/java/eu/stratosphere/streaming/api/streamrecord/MyGeneric.java
已删除
100755 → 0
浏览文件 @
a1116c3f
package
eu.stratosphere.streaming.api.streamrecord
;
import
java.io.Serializable
;
public
abstract
class
MyGeneric
<
IN
>
implements
Serializable
{
public
abstract
void
asd
();
}
flink-addons/flink-streaming/src/test/java/eu/stratosphere/streaming/api/streamrecord/MyGeneric2.java
已删除
100755 → 0
浏览文件 @
a1116c3f
package
eu.stratosphere.streaming.api.streamrecord
;
public
class
MyGeneric2
extends
MyGeneric
<
Integer
>{
@Override
public
void
asd
()
{
// TODO Auto-generated method stub
}
}
flink-addons/flink-streaming/src/test/java/eu/stratosphere/streaming/api/streamrecord/StreamRecordTest.java
浏览文件 @
a3b1dfa0
...
...
@@ -15,60 +15,479 @@
package
eu.stratosphere.streaming.api.streamrecord
;
import
static
org
.
junit
.
Assert
.
assertArrayEquals
;
import
static
org
.
junit
.
Assert
.
assertEquals
;
import
static
org
.
junit
.
Assert
.
assertFalse
;
import
static
org
.
junit
.
Assert
.
assertTrue
;
import
static
org
.
junit
.
Assert
.
fail
;
import
java.io.ByteArrayInputStream
;
import
java.io.ByteArrayOutputStream
;
import
java.io.DataInputStream
;
import
java.io.DataOutputStream
;
import
java.io.IOException
;
import
java.io.ObjectInputStream
;
import
java.io.ObjectOutputStream
;
import
java.util.ArrayList
;
import
java.util.Arrays
;
import
java.util.LinkedList
;
import
java.util.List
;
import
org.apache.commons.math.stat.regression.OLSMultipleLinearRegression
;
import
org.junit.Test
;
import
eu.stratosphere.api.java.tuple.Tuple
;
import
eu.stratosphere.api.java.tuple.Tuple1
;
import
eu.stratosphere.api.java.tuple.Tuple2
;
import
eu.stratosphere.api.java.tuple.Tuple3
;
import
eu.stratosphere.api.java.tuple.Tuple5
;
import
eu.stratosphere.api.java.tuple.Tuple9
;
import
eu.stratosphere.api.java.typeutils.TupleTypeInfo
;
import
eu.stratosphere.api.java.typeutils.TypeExtractor
;
import
eu.stratosphere.types.TypeInformation
;
public
class
StreamRecordTest
{
@Test
public
void
constructorTest
()
{
StreamRecord
record
=
new
StreamRecord
(
10
);
assertEquals
(
10
,
record
.
getBatchSize
());
public
void
singleRecordSetGetTest
()
{
StreamRecord
record
=
new
StreamRecord
(
new
Tuple9
<
String
,
Integer
,
Long
,
Boolean
,
Double
,
Byte
,
Character
,
Float
,
Short
>(
"Stratosphere"
,
1
,
2L
,
true
,
3.5
,
(
byte
)
0xa
,
'a'
,
0.1f
,
(
short
)
42
));
List
<
Tuple
>
tuples
=
new
ArrayList
<
Tuple
>();
tuples
.
add
(
new
Tuple1
<
Integer
>(
2
));
tuples
.
add
(
new
Tuple1
<
Integer
>(
3
));
assertEquals
(
9
,
record
.
getNumOfFields
());
assertEquals
(
1
,
record
.
getNumOfTuples
());
StreamRecord
record1
=
new
StreamRecord
(
tuples
);
assertEquals
(
"Stratosphere"
,
record
.
getString
(
0
));
assertEquals
((
Integer
)
1
,
record
.
getInteger
(
1
));
assertEquals
((
Long
)
2L
,
record
.
getLong
(
2
));
assertEquals
(
true
,
record
.
getBoolean
(
3
));
assertEquals
((
Double
)
3.5
,
record
.
getDouble
(
4
));
assertEquals
((
Byte
)
(
byte
)
0xa
,
record
.
getByte
(
5
));
assertEquals
((
Character
)
'a'
,
record
.
getCharacter
(
6
));
assertEquals
((
Float
)
0.1f
,
record
.
getFloat
(
7
));
assertEquals
((
Short
)
(
short
)
42
,
record
.
getShort
(
8
));
assertEquals
(
2
,
record1
.
getBatchSize
());
Tuple9
<
String
,
Integer
,
Long
,
Boolean
,
Double
,
Byte
,
Character
,
Float
,
Short
>
tuple
=
new
Tuple9
<
String
,
Integer
,
Long
,
Boolean
,
Double
,
Byte
,
Character
,
Float
,
Short
>();
record
.
getTupleInto
(
tuple
);
assertEquals
(
"Stratosphere"
,
tuple
.
getField
(
0
));
assertEquals
((
Integer
)
1
,
tuple
.
getField
(
1
));
assertEquals
((
Long
)
2L
,
tuple
.
getField
(
2
));
assertEquals
(
true
,
tuple
.
getField
(
3
));
assertEquals
((
Double
)
3.5
,
tuple
.
getField
(
4
));
assertEquals
((
Byte
)
(
byte
)
0xa
,
tuple
.
getField
(
5
));
assertEquals
((
Character
)
'a'
,
tuple
.
getField
(
6
));
assertEquals
((
Float
)
0.1f
,
tuple
.
getField
(
7
));
assertEquals
((
Short
)
(
short
)
42
,
tuple
.
getField
(
8
));
record
.
setString
(
0
,
"Streaming"
);
record
.
setInteger
(
1
,
2
);
record
.
setLong
(
2
,
3L
);
record
.
setBoolean
(
3
,
false
);
record
.
setDouble
(
4
,
4.5
);
record
.
setByte
(
5
,
(
byte
)
0xb
);
record
.
setCharacter
(
6
,
'b'
);
record
.
setFloat
(
7
,
0.2f
);
record
.
setShort
(
8
,
(
short
)
69
);
assertEquals
(
"Streaming"
,
record
.
getString
(
0
));
assertEquals
((
Integer
)
2
,
record
.
getInteger
(
1
));
assertEquals
((
Long
)
3L
,
record
.
getLong
(
2
));
assertEquals
(
false
,
record
.
getBoolean
(
3
));
assertEquals
((
Double
)
4.5
,
record
.
getDouble
(
4
));
assertEquals
((
Byte
)
(
byte
)
0xb
,
record
.
getByte
(
5
));
assertEquals
((
Character
)
'b'
,
record
.
getCharacter
(
6
));
assertEquals
((
Float
)
0.2f
,
record
.
getFloat
(
7
));
assertEquals
((
Short
)
(
short
)
69
,
record
.
getShort
(
8
));
record
.
setString
(
0
,
0
,
""
);
record
.
setInteger
(
0
,
1
,
0
);
record
.
setLong
(
0
,
2
,
0L
);
record
.
setBoolean
(
0
,
3
,
false
);
record
.
setDouble
(
0
,
4
,
0
.);
record
.
setByte
(
0
,
5
,
(
byte
)
0x0
);
record
.
setCharacter
(
0
,
6
,
'\0'
);
record
.
setFloat
(
0
,
7
,
0
.
f
);
record
.
setShort
(
0
,
8
,
(
short
)
0
);
assertEquals
(
""
,
record
.
getString
(
0
,
0
));
assertEquals
((
Integer
)
0
,
record
.
getInteger
(
0
,
1
));
assertEquals
((
Long
)
0L
,
record
.
getLong
(
0
,
2
));
assertEquals
(
false
,
record
.
getBoolean
(
0
,
3
));
assertEquals
((
Double
)
0
.,
record
.
getDouble
(
0
,
4
));
assertEquals
((
Byte
)
(
byte
)
0x0
,
record
.
getByte
(
0
,
5
));
assertEquals
((
Character
)
'\0'
,
record
.
getCharacter
(
0
,
6
));
assertEquals
((
Float
)
0
.
f
,
record
.
getFloat
(
0
,
7
));
assertEquals
((
Short
)
(
short
)
0
,
record
.
getShort
(
0
,
8
));
}
@Test
public
void
batchRecordSetGetTest
()
{
StreamRecord
record
=
new
StreamRecord
(
5
,
2
);
Tuple5
<
String
,
Integer
,
Long
,
Boolean
,
Double
>
tuple
=
new
Tuple5
<
String
,
Integer
,
Long
,
Boolean
,
Double
>(
"Stratosphere"
,
1
,
2L
,
true
,
3.5
);
record
.
addTuple
(
StreamRecord
.
copyTuple
(
tuple
));
tuple
.
setField
(
""
,
0
);
tuple
.
setField
(
0
,
1
);
tuple
.
setField
(
0L
,
2
);
tuple
.
setField
(
false
,
3
);
tuple
.
setField
(
0
.,
4
);
record
.
addTuple
(
tuple
);
try
{
record
.
addTuple
(
new
Tuple1
<
String
>(
"4"
));
fail
();
}
catch
(
TupleSizeMismatchException
e
)
{
}
assertEquals
(
5
,
record
.
getNumOfFields
());
assertEquals
(
2
,
record
.
getNumOfTuples
());
assertEquals
(
"Stratosphere"
,
record
.
getString
(
0
,
0
));
assertEquals
((
Integer
)
1
,
record
.
getInteger
(
0
,
1
));
assertEquals
((
Long
)
2L
,
record
.
getLong
(
0
,
2
));
assertEquals
(
true
,
record
.
getBoolean
(
0
,
3
));
assertEquals
((
Double
)
3.5
,
record
.
getDouble
(
0
,
4
));
assertEquals
(
""
,
record
.
getString
(
1
,
0
));
assertEquals
((
Integer
)
0
,
record
.
getInteger
(
1
,
1
));
assertEquals
((
Long
)
0L
,
record
.
getLong
(
1
,
2
));
assertEquals
(
false
,
record
.
getBoolean
(
1
,
3
));
assertEquals
((
Double
)
0
.,
record
.
getDouble
(
1
,
4
));
record
.
setTuple
(
new
Tuple5
<
String
,
Integer
,
Long
,
Boolean
,
Double
>(
""
,
0
,
0L
,
false
,
0
.));
assertEquals
(
5
,
record
.
getNumOfFields
());
assertEquals
(
2
,
record
.
getNumOfTuples
());
assertEquals
(
""
,
record
.
getString
(
0
,
0
));
assertEquals
((
Integer
)
0
,
record
.
getInteger
(
0
,
1
));
assertEquals
((
Long
)
0L
,
record
.
getLong
(
0
,
2
));
assertEquals
(
false
,
record
.
getBoolean
(
0
,
3
));
assertEquals
((
Double
)
0
.,
record
.
getDouble
(
0
,
4
));
record
.
setTuple
(
1
,
new
Tuple5
<
String
,
Integer
,
Long
,
Boolean
,
Double
>(
"Stratosphere"
,
1
,
2L
,
true
,
3.5
));
assertEquals
(
"Stratosphere"
,
record
.
getString
(
1
,
0
));
assertEquals
((
Integer
)
1
,
record
.
getInteger
(
1
,
1
));
assertEquals
((
Long
)
2L
,
record
.
getLong
(
1
,
2
));
assertEquals
(
true
,
record
.
getBoolean
(
1
,
3
));
assertEquals
((
Double
)
3.5
,
record
.
getDouble
(
1
,
4
));
record
.
removeTuple
(
1
);
assertEquals
(
1
,
record
.
getNumOfTuples
());
assertEquals
(
""
,
record
.
getString
(
0
,
0
));
assertEquals
((
Integer
)
0
,
record
.
getInteger
(
0
,
1
));
assertEquals
((
Long
)
0L
,
record
.
getLong
(
0
,
2
));
assertEquals
(
false
,
record
.
getBoolean
(
0
,
3
));
assertEquals
((
Double
)
0
.,
record
.
getDouble
(
0
,
4
));
record
.
addTuple
(
0
,
new
Tuple5
<
String
,
Integer
,
Long
,
Boolean
,
Double
>(
"Stratosphere"
,
1
,
2L
,
true
,
3.5
));
assertEquals
(
2
,
record
.
getNumOfTuples
());
assertEquals
(
"Stratosphere"
,
record
.
getString
(
0
,
0
));
assertEquals
((
Integer
)
1
,
record
.
getInteger
(
0
,
1
));
assertEquals
((
Long
)
2L
,
record
.
getLong
(
0
,
2
));
assertEquals
(
true
,
record
.
getBoolean
(
0
,
3
));
assertEquals
((
Double
)
3.5
,
record
.
getDouble
(
0
,
4
));
}
@Test
public
void
copyTest
()
throws
IOException
{
StreamRecord
a
=
new
StreamRecord
(
new
Tuple1
<
String
>(
"Big"
));
a
.
setId
(
0
);
StreamRecord
b
=
a
.
copy
();
assertTrue
(
a
.
getField
(
0
).
equals
(
b
.
getField
(
0
)));
assertTrue
(
a
.
getId
().
equals
(
b
.
getId
()));
b
.
setId
(
2
);
b
.
setTuple
(
new
Tuple1
<
String
>(
"Data"
));
assertFalse
(
a
.
getId
().
equals
(
b
.
getId
()));
assertFalse
(
a
.
getField
(
0
).
equals
(
b
.
getField
(
0
)));
final
int
ITERATION
=
10000
;
StreamRecord
c
=
new
StreamRecord
(
new
Tuple1
<
String
>(
"Big"
));
long
t
=
System
.
nanoTime
();
for
(
int
i
=
0
;
i
<
ITERATION
;
i
++)
{
c
.
copySerialized
();
}
long
t2
=
System
.
nanoTime
()
-
t
;
System
.
out
.
println
(
"Serialized copy:\t"
+
t2
+
" ns"
);
t
=
System
.
nanoTime
();
for
(
int
i
=
0
;
i
<
ITERATION
;
i
++)
{
c
.
copy
();
}
t2
=
System
.
nanoTime
()
-
t
;
System
.
out
.
println
(
"Copy:\t"
+
t2
+
" ns"
);
StreamRecord
record2
=
new
StreamRecord
(
record1
);
assertEquals
(
2
,
record2
.
getBatchSize
());
}
@Test
public
void
typeExtractTest
()
throws
IOException
,
ClassNotFoundException
{
public
void
exceptionTest
()
{
StreamRecord
a
=
new
StreamRecord
(
new
Tuple1
<
String
>(
"Big"
));
try
{
a
.
setTuple
(
4
,
new
Tuple1
<
String
>(
"Data"
));
fail
();
}
catch
(
NoSuchTupleException
e
)
{
}
try
{
a
.
setTuple
(
new
Tuple2
<
String
,
String
>(
"Data"
,
"Stratosphere"
));
fail
();
}
catch
(
TupleSizeMismatchException
e
)
{
}
StreamRecord
b
=
new
StreamRecord
();
try
{
b
.
addTuple
(
new
Tuple2
<
String
,
String
>(
"Data"
,
"Stratosphere"
));
fail
();
}
catch
(
TupleSizeMismatchException
e
)
{
}
try
{
a
.
getField
(
3
);
fail
();
}
catch
(
NoSuchFieldException
e
)
{
}
try
{
a
.
getBoolean
(
0
);
fail
();
}
catch
(
FieldTypeMismatchException
e
)
{}
try
{
a
.
getByte
(
0
);
fail
();
}
catch
(
FieldTypeMismatchException
e
)
{}
try
{
a
.
getCharacter
(
0
);
fail
();
}
catch
(
FieldTypeMismatchException
e
)
{}
try
{
a
.
getDouble
(
0
);
fail
();
}
catch
(
FieldTypeMismatchException
e
)
{}
try
{
a
.
getFloat
(
0
);
fail
();
}
catch
(
FieldTypeMismatchException
e
)
{}
try
{
a
.
getInteger
(
0
);
fail
();
}
catch
(
FieldTypeMismatchException
e
)
{}
try
{
a
.
getLong
(
0
);
fail
();
}
catch
(
FieldTypeMismatchException
e
)
{}
try
{
a
.
getShort
(
0
);
fail
();
}
catch
(
FieldTypeMismatchException
e
)
{}
StreamRecord
c
=
new
StreamRecord
(
new
Tuple1
<
Integer
>(
1
));
try
{
c
.
getString
(
0
);
fail
();
}
catch
(
FieldTypeMismatchException
e
)
{}
}
@Test
public
void
writeReadTest
()
{
ByteArrayOutputStream
buff
=
new
ByteArrayOutputStream
();
ObjectOutputStream
out
=
new
Object
OutputStream
(
buff
);
DataOutputStream
out
=
new
Data
OutputStream
(
buff
);
MyGeneric
<?>
g
=
new
MyGeneric2
();
out
.
writeObject
(
g
);
int
num
=
42
;
String
str
=
"above clouds"
;
Integer
[]
intArray
=
new
Integer
[]
{
1
,
2
};
Tuple3
<
Integer
,
String
,
Integer
[]>
tuple1
=
new
Tuple3
<
Integer
,
String
,
Integer
[]>(
num
,
str
,
intArray
);
Tuple3
<
Integer
,
String
,
Integer
[]>
tuple2
=
new
Tuple3
<
Integer
,
String
,
Integer
[]>(
1
,
""
,
new
Integer
[]
{
1
,
2
});
StreamRecord
rec
=
new
StreamRecord
(
tuple1
);
rec
.
addTuple
(
tuple2
);
ObjectInputStream
in
=
new
ObjectInputStream
(
new
ByteArrayInputStream
(
buff
.
toByteArray
()));
try
{
rec
.
write
(
out
);
DataInputStream
in
=
new
DataInputStream
(
new
ByteArrayInputStream
(
buff
.
toByteArray
()));
MyGeneric
<?>
f
=
(
MyGeneric
<?>)
in
.
readObject
();
StreamRecord
newRec
=
new
StreamRecord
();
newRec
.
read
(
in
);
TypeInformation
<?>
ti
=
TypeExtractor
.
createTypeInfo
(
MyGeneric
.
class
,
f
.
getClass
(),
0
,
null
,
null
);
assertEquals
(
2
,
newRec
.
getNumOfTuples
());
System
.
out
.
println
(
"Type info: "
+
ti
);
@SuppressWarnings
(
"unchecked"
)
Tuple3
<
Integer
,
String
,
Integer
[]>
tupleOut1
=
(
Tuple3
<
Integer
,
String
,
Integer
[]>)
newRec
.
getTuple
(
0
);
assertEquals
(
tupleOut1
.
getField
(
0
),
42
);
assertEquals
(
str
,
tupleOut1
.
getField
(
1
));
assertArrayEquals
(
intArray
,
(
Integer
[])
tupleOut1
.
getField
(
2
));
@SuppressWarnings
(
"unchecked"
)
Tuple3
<
Integer
,
String
,
Integer
[]>
tupleOut2
=
(
Tuple3
<
Integer
,
String
,
Integer
[]>)
newRec
.
getTuple
(
1
);
assertEquals
(
tupleOut2
.
getField
(
0
),
1
);
assertEquals
(
""
,
tupleOut2
.
getField
(
1
));
assertArrayEquals
(
new
Integer
[]
{
1
,
2
},
(
Integer
[])
tupleOut2
.
getField
(
2
));
}
catch
(
IOException
e
)
{
fail
();
e
.
printStackTrace
();
}
}
@Test
public
void
tupleCopyTest
()
{
Tuple3
<
String
,
Integer
,
Double
[]>
t1
=
new
Tuple3
<
String
,
Integer
,
Double
[]>(
"a"
,
1
,
new
Double
[]
{
4.2
});
@SuppressWarnings
(
"rawtypes"
)
Tuple3
t2
=
(
Tuple3
)
StreamRecord
.
copyTuple
(
t1
);
assertEquals
(
"a"
,
t2
.
getField
(
0
));
assertEquals
(
1
,
t2
.
getField
(
1
));
assertArrayEquals
(
new
Double
[]
{
4.2
},
(
Double
[])
t2
.
getField
(
2
));
t1
.
setField
(
2
,
1
);
assertEquals
(
1
,
t2
.
getField
(
1
));
assertEquals
(
2
,
t1
.
getField
(
1
));
t1
.
setField
(
new
Double
[]
{
3.14
},
2
);
assertArrayEquals
(
new
Double
[]
{
3.14
},
(
Double
[])
t1
.
getField
(
2
));
assertArrayEquals
(
new
Double
[]
{
4.2
},
(
Double
[])
t2
.
getField
(
2
));
assertEquals
(
t1
.
getField
(
0
).
getClass
(),
t2
.
getField
(
0
).
getClass
());
assertEquals
(
t1
.
getField
(
1
).
getClass
(),
t2
.
getField
(
1
).
getClass
());
}
@Test
public
void
tupleArraySerializationTest
()
throws
IOException
{
Tuple9
<
Boolean
[],
Byte
[],
Character
[],
Double
[],
Float
[],
Integer
[],
Long
[],
Short
[],
String
[]>
t1
=
new
Tuple9
<
Boolean
[],
Byte
[],
Character
[],
Double
[],
Float
[],
Integer
[],
Long
[],
Short
[],
String
[]>(
new
Boolean
[]
{
true
},
new
Byte
[]
{
12
},
new
Character
[]
{
'a'
},
new
Double
[]
{
12.5
},
new
Float
[]
{
13.5f
},
new
Integer
[]
{
1234
},
new
Long
[]
{
12345678900
l
},
new
Short
[]
{
12345
},
new
String
[]
{
"something"
});
StreamRecord
s1
=
new
StreamRecord
(
t1
);
StreamRecord
s2
=
s1
.
copySerialized
();
@SuppressWarnings
(
"rawtypes"
)
Tuple9
t2
=
(
Tuple9
)
s2
.
getTuple
();
assertArrayEquals
(
new
Boolean
[]
{
true
},
(
Boolean
[])
t2
.
getField
(
0
));
assertArrayEquals
(
new
Byte
[]
{
12
},
(
Byte
[])
t2
.
getField
(
1
));
assertArrayEquals
(
new
Character
[]
{
'a'
},
(
Character
[])
t2
.
getField
(
2
));
assertArrayEquals
(
new
Double
[]
{
12.5
},
(
Double
[])
t2
.
getField
(
3
));
assertArrayEquals
(
new
Float
[]
{
13.5f
},
(
Float
[])
t2
.
getField
(
4
));
assertArrayEquals
(
new
Integer
[]
{
1234
},
(
Integer
[])
t2
.
getField
(
5
));
assertArrayEquals
(
new
Long
[]
{
12345678900
l
},
(
Long
[])
t2
.
getField
(
6
));
assertArrayEquals
(
new
Short
[]
{
12345
},
(
Short
[])
t2
.
getField
(
7
));
assertArrayEquals
(
new
String
[]
{
"something"
},
(
String
[])
t2
.
getField
(
8
));
assertEquals
(
t1
.
getField
(
0
).
getClass
(),
t2
.
getField
(
0
).
getClass
());
assertEquals
(
t1
.
getField
(
1
).
getClass
(),
t2
.
getField
(
1
).
getClass
());
assertEquals
(
t1
.
getField
(
2
).
getClass
(),
t2
.
getField
(
2
).
getClass
());
assertEquals
(
t1
.
getField
(
3
).
getClass
(),
t2
.
getField
(
3
).
getClass
());
assertEquals
(
t1
.
getField
(
4
).
getClass
(),
t2
.
getField
(
4
).
getClass
());
assertEquals
(
t1
.
getField
(
5
).
getClass
(),
t2
.
getField
(
5
).
getClass
());
assertEquals
(
t1
.
getField
(
6
).
getClass
(),
t2
.
getField
(
6
).
getClass
());
assertEquals
(
t1
.
getField
(
7
).
getClass
(),
t2
.
getField
(
7
).
getClass
());
assertEquals
(
t1
.
getField
(
8
).
getClass
(),
t2
.
getField
(
8
).
getClass
());
}
// TODO:measure performance of different serialization logics
@Test
public
void
typeCopyTest
()
throws
NoSuchTupleException
,
IOException
{
StreamRecord
rec
=
new
StreamRecord
(
new
Tuple9
<
Boolean
,
Byte
,
Character
,
Double
,
Float
,
Integer
,
Long
,
Short
,
String
>(
(
Boolean
)
true
,
(
Byte
)
(
byte
)
12
,
(
Character
)
'a'
,
(
Double
)
12.5
,
(
Float
)
(
float
)
13.5
,
(
Integer
)
1234
,
(
Long
)
12345678900
l
,
(
Short
)
(
short
)
12345
,
"something"
));
ByteArrayOutputStream
buff3
=
new
ByteArrayOutputStream
();
DataOutputStream
out3
=
new
DataOutputStream
(
buff3
);
for
(
int
i
=
0
;
i
<
1000
;
i
++)
{
out3
.
write
(
rec
.
tupleTypesToByteArray
(
rec
.
getTuple
()));
}
}
@Test
public
void
typeArrayCopyTest
()
throws
NoSuchTupleException
,
IOException
{
StreamRecord
rec
=
new
StreamRecord
(
new
Tuple9
<
Boolean
[],
Byte
[],
Character
[],
Double
[],
Float
[],
Integer
[],
Long
[],
Short
[],
String
[]>(
new
Boolean
[]
{
true
},
new
Byte
[]
{
12
},
new
Character
[]
{
'a'
},
new
Double
[]
{
12.5
},
new
Float
[]
{
13.5f
},
new
Integer
[]
{
1234
},
new
Long
[]
{
12345678900
l
},
new
Short
[]
{
12345
},
new
String
[]
{
"something"
}));
ByteArrayOutputStream
buff
=
new
ByteArrayOutputStream
();
DataOutputStream
out
=
new
DataOutputStream
(
buff
);
for
(
int
i
=
0
;
i
<
10000
;
i
++)
{
out
.
write
(
rec
.
tupleTypesToByteArray
(
rec
.
getTuple
()));
}
DataInputStream
in
=
new
DataInputStream
(
new
ByteArrayInputStream
(
buff
.
toByteArray
()));
StreamRecord
rec2
=
new
StreamRecord
();
Long
start
=
System
.
nanoTime
();
for
(
int
i
=
0
;
i
<
10000
;
i
++)
{
byte
[]
byteTypes
=
new
byte
[
9
];
in
.
read
(
byteTypes
);
TypeInformation
<?>[]
basicTypes
=
rec2
.
tupleTypesFromByteArray
(
byteTypes
);
@SuppressWarnings
(
"unused"
)
TupleTypeInfo
<
Tuple
>
typeInfo
=
new
TupleTypeInfo
<
Tuple
>(
basicTypes
);
}
System
.
out
.
println
(
"Type copy with ByteArray:\t"
+
(
System
.
nanoTime
()
-
start
)
+
" ns"
);
start
=
System
.
nanoTime
();
byte
[]
byteTypes
=
rec
.
tupleTypesToByteArray
(
rec
.
getTuple
());
Tuple
t
=
rec
.
getTuple
();
start
=
System
.
nanoTime
();
for
(
int
i
=
0
;
i
<
10000
;
i
++)
{
// rec2.tupleBasicTypesFromByteArray(byteTypes, 9);
TypeInformation
<?>[]
basicTypes
=
rec2
.
tupleTypesFromByteArray
(
byteTypes
);
@SuppressWarnings
(
"unused"
)
TupleTypeInfo
<
Tuple
>
typeInfo
=
new
TupleTypeInfo
<
Tuple
>(
basicTypes
);
}
System
.
out
.
println
(
"Write with infoArray:\t\t"
+
(
System
.
nanoTime
()
-
start
)
+
" ns"
);
start
=
System
.
nanoTime
();
for
(
int
i
=
0
;
i
<
10000
;
i
++)
{
@SuppressWarnings
(
"unused"
)
TupleTypeInfo
<
Tuple
>
typeInfo
=
(
TupleTypeInfo
<
Tuple
>)
TypeExtractor
.
getForObject
(
t
);
}
System
.
out
.
println
(
"Write with extract:\t\t"
+
(
System
.
nanoTime
()
-
start
)
+
" ns"
);
}
@Test
public
void
batchIteratorTest
()
{
List
<
Tuple
>
tupleList
=
new
LinkedList
<
Tuple
>();
tupleList
.
add
(
new
Tuple1
<
String
>(
"Big"
));
tupleList
.
add
(
new
Tuple1
<
String
>(
"Data"
));
StreamRecord
a
=
new
StreamRecord
(
tupleList
);
assertEquals
(
2
,
a
.
getNumOfTuples
());
assertEquals
(
1
,
a
.
getNumOfFields
());
for
(
Tuple
t
:
a
.
getBatchIterable
())
{
System
.
out
.
println
(
t
);
}
OLSMultipleLinearRegression
ols
=
new
OLSMultipleLinearRegression
();
ols
.
newSampleData
(
new
double
[]
{
1.0
,
2.0
},
new
double
[][]
{
{
1
,
2
},
{
3
,
4
}
});
System
.
out
.
println
(
Arrays
.
toString
(
ols
.
estimateRegressionParameters
()));
}
}
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录