Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
doujutun3207
flink
提交
2d17a6cd
F
flink
项目概览
doujutun3207
/
flink
与 Fork 源项目一致
从无法访问的项目Fork
通知
24
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
F
flink
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
2d17a6cd
编写于
7月 14, 2014
作者:
G
gyfora
提交者:
Stephan Ewen
8月 18, 2014
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
[streaming] StreamRecord serialization improvement
上级
53e54a60
变更
2
隐藏空白更改
内联
并排
Showing
2 changed file
with
45 addition
and
57 deletion
+45
-57
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/api/streamrecord/StreamRecord.java
...stratosphere/streaming/api/streamrecord/StreamRecord.java
+24
-51
flink-addons/flink-streaming/src/test/java/eu/stratosphere/streaming/api/streamrecord/StreamRecordTest.java
...tosphere/streaming/api/streamrecord/StreamRecordTest.java
+21
-6
未找到文件。
flink-addons/flink-streaming/src/main/java/eu/stratosphere/streaming/api/streamrecord/StreamRecord.java
浏览文件 @
2d17a6cd
...
...
@@ -1153,55 +1153,6 @@ public class StreamRecord implements IOReadableWritable, Serializable {
return
basicTypes
;
}
/**
* Writes tuple to the specified DataOutput
*
* @param tuple
* Tuple to be written
* @param out
* Output chosen
* @throws IOException
*/
private
void
writeTuple
(
Tuple
tuple
,
DataOutput
out
)
throws
IOException
{
// TODO: exception for empty record - no getField
byte
[]
typesInByte
=
tupleTypesToByteArray
(
getTuple
());
TupleTypeInfo
<
Tuple
>
typeInfo
=
new
TupleTypeInfo
<
Tuple
>(
tupleTypesFromByteArray
(
typesInByte
));
TupleSerializer
<
Tuple
>
tupleSerializer
=
(
TupleSerializer
<
Tuple
>)
typeInfo
.
createSerializer
();
SerializationDelegate
<
Tuple
>
serializationDelegate
=
new
SerializationDelegate
<
Tuple
>(
tupleSerializer
);
serializationDelegate
.
setInstance
(
tuple
);
out
.
write
(
typesInByte
);
serializationDelegate
.
write
(
out
);
}
/**
* Reads a tuple from the specified DataInput
*
* @param in
* Input chosen
* @return Tuple read
* @throws IOException
*/
private
Tuple
readTuple
(
DataInput
in
,
int
numberOfFields
)
throws
IOException
{
byte
[]
typesInByte
=
new
byte
[
numberOfFields
];
in
.
readFully
(
typesInByte
,
0
,
numberOfFields
);
TupleTypeInfo
<
Tuple
>
typeInfo
=
new
TupleTypeInfo
<
Tuple
>(
tupleTypesFromByteArray
(
typesInByte
));
TupleSerializer
<
Tuple
>
tupleSerializer
=
typeInfo
.
createSerializer
();
DeserializationDelegate
<
Tuple
>
dd
=
new
DeserializationDelegate
<
Tuple
>(
tupleSerializer
);
dd
.
setInstance
(
tupleSerializer
.
createInstance
());
dd
.
read
(
in
);
return
dd
.
getInstance
();
}
/**
* Write method definition for the IOReadableWritable interface
*/
...
...
@@ -1213,8 +1164,19 @@ public class StreamRecord implements IOReadableWritable, Serializable {
out
.
writeByte
(
numOfFields
);
out
.
writeInt
(
numOfTuples
);
byte
[]
typesInByte
=
tupleTypesToByteArray
(
getTuple
());
out
.
write
(
typesInByte
);
TupleTypeInfo
<
Tuple
>
typeInfo
=
new
TupleTypeInfo
<
Tuple
>(
tupleTypesFromByteArray
(
typesInByte
));
TupleSerializer
<
Tuple
>
tupleSerializer
=
(
TupleSerializer
<
Tuple
>)
typeInfo
.
createSerializer
();
SerializationDelegate
<
Tuple
>
serializationDelegate
=
new
SerializationDelegate
<
Tuple
>(
tupleSerializer
);
for
(
Tuple
tuple
:
tupleBatch
)
{
writeTuple
(
tuple
,
out
);
serializationDelegate
.
setInstance
(
tuple
);
serializationDelegate
.
write
(
out
);
}
}
...
...
@@ -1231,8 +1193,19 @@ public class StreamRecord implements IOReadableWritable, Serializable {
tupleBatch
=
new
ArrayList
<
Tuple
>(
numOfTuples
);
byte
[]
typesInByte
=
new
byte
[
numOfFields
];
in
.
readFully
(
typesInByte
,
0
,
numOfFields
);
TupleTypeInfo
<
Tuple
>
typeInfo
=
new
TupleTypeInfo
<
Tuple
>(
tupleTypesFromByteArray
(
typesInByte
));
TupleSerializer
<
Tuple
>
tupleSerializer
=
typeInfo
.
createSerializer
();
DeserializationDelegate
<
Tuple
>
dd
=
new
DeserializationDelegate
<
Tuple
>(
tupleSerializer
);
for
(
int
k
=
0
;
k
<
numOfTuples
;
++
k
)
{
tupleBatch
.
add
(
readTuple
(
in
,
numOfFields
));
dd
.
setInstance
(
tupleSerializer
.
createInstance
());
dd
.
read
(
in
);
tupleBatch
.
add
(
dd
.
getInstance
());
}
}
...
...
flink-addons/flink-streaming/src/test/java/eu/stratosphere/streaming/api/streamrecord/StreamRecordTest.java
浏览文件 @
2d17a6cd
...
...
@@ -264,8 +264,12 @@ public class StreamRecordTest {
int
num
=
42
;
String
str
=
"above clouds"
;
Integer
[]
intArray
=
new
Integer
[]
{
1
,
2
};
StreamRecord
rec
=
new
StreamRecord
(
new
Tuple3
<
Integer
,
String
,
Integer
[]>(
num
,
str
,
intArray
));
Tuple3
<
Integer
,
String
,
Integer
[]>
tuple1
=
new
Tuple3
<
Integer
,
String
,
Integer
[]>(
num
,
str
,
intArray
);
Tuple3
<
Integer
,
String
,
Integer
[]>
tuple2
=
new
Tuple3
<
Integer
,
String
,
Integer
[]>(
1
,
""
,
new
Integer
[]
{
1
,
2
});
StreamRecord
rec
=
new
StreamRecord
(
tuple1
);
rec
.
addTuple
(
tuple2
);
try
{
rec
.
write
(
out
);
...
...
@@ -273,13 +277,24 @@ public class StreamRecordTest {
StreamRecord
newRec
=
new
StreamRecord
();
newRec
.
read
(
in
);
assertEquals
(
2
,
newRec
.
getNumOfTuples
());
@SuppressWarnings
(
"unchecked"
)
Tuple3
<
Integer
,
String
,
Integer
[]>
tupleOut
=
(
Tuple3
<
Integer
,
String
,
Integer
[]>)
newRec
Tuple3
<
Integer
,
String
,
Integer
[]>
tupleOut
1
=
(
Tuple3
<
Integer
,
String
,
Integer
[]>)
newRec
.
getTuple
(
0
);
assertEquals
(
tupleOut
.
getField
(
0
),
42
);
assertEquals
(
str
,
tupleOut
.
getField
(
1
));
assertArrayEquals
(
intArray
,
(
Integer
[])
tupleOut
.
getField
(
2
));
assertEquals
(
tupleOut1
.
getField
(
0
),
42
);
assertEquals
(
str
,
tupleOut1
.
getField
(
1
));
assertArrayEquals
(
intArray
,
(
Integer
[])
tupleOut1
.
getField
(
2
));
@SuppressWarnings
(
"unchecked"
)
Tuple3
<
Integer
,
String
,
Integer
[]>
tupleOut2
=
(
Tuple3
<
Integer
,
String
,
Integer
[]>)
newRec
.
getTuple
(
1
);
assertEquals
(
tupleOut2
.
getField
(
0
),
1
);
assertEquals
(
""
,
tupleOut2
.
getField
(
1
));
assertArrayEquals
(
new
Integer
[]
{
1
,
2
},
(
Integer
[])
tupleOut2
.
getField
(
2
));
}
catch
(
IOException
e
)
{
fail
();
e
.
printStackTrace
();
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录