Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
doujutun3207
flink
提交
06503c8f
F
flink
项目概览
doujutun3207
/
flink
与 Fork 源项目一致
从无法访问的项目Fork
通知
24
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
F
flink
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
06503c8f
编写于
1月 07, 2015
作者:
G
Gyula Fora
提交者:
mbalassi
1月 08, 2015
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
[FLINK-1367] [scala] [streaming] Field aggregations added to streaming scala api
上级
10a81862
变更
4
隐藏空白更改
内联
并排
Showing
4 changed file
with
120 addition
and
20 deletion
+120
-20
flink-addons/flink-streaming/flink-streaming-examples/src/main/scala/org/apache/flink/streaming/scala/examples/windowing/TopSpeedWindowing.scala
...treaming/scala/examples/windowing/TopSpeedWindowing.scala
+1
-1
flink-addons/flink-streaming/flink-streaming-scala/src/main/scala/org/apache/flink/streaming/api/scala/DataStream.scala
...ala/org/apache/flink/streaming/api/scala/DataStream.scala
+52
-13
flink-addons/flink-streaming/flink-streaming-scala/src/main/scala/org/apache/flink/streaming/api/scala/WindowedDataStream.scala
...apache/flink/streaming/api/scala/WindowedDataStream.scala
+47
-6
flink-addons/flink-streaming/flink-streaming-scala/src/main/scala/org/apache/flink/streaming/api/scala/package.scala
.../scala/org/apache/flink/streaming/api/scala/package.scala
+20
-0
未找到文件。
flink-addons/flink-streaming/flink-streaming-examples/src/main/scala/org/apache/flink/streaming/scala/examples/windowing/TopSpeedWindowing.scala
浏览文件 @
06503c8f
...
...
@@ -50,7 +50,7 @@ object TopSpeedWindowing {
.
window
(
Time
.
of
(
evictionSec
,
SECONDS
))
.
every
(
Delta
.
of
[
CarEvent
](
triggerMeters
,
(
oldSp
,
newSp
)
=>
newSp
.
distance
-
oldSp
.
distance
,
CarEvent
(
0
,
0
,
0
,
0
)))
.
reduce
((
x
,
y
)
=>
if
(
x
.
speed
>
y
.
speed
)
x
else
y
)
.
maxBy
(
"speed"
)
cars
print
...
...
flink-addons/flink-streaming/flink-streaming-scala/src/main/scala/org/apache/flink/streaming/api/scala/DataStream.scala
浏览文件 @
06503c8f
...
...
@@ -138,7 +138,7 @@ class DataStream[T](javaStream: JavaStream[T]) {
/**
* Sets the partitioning of the DataStream so that the output values all go to
* the first instance of the next processing operator. Use this setting with care
* since it might cause a serious performance bottlenec
t
in the application.
* since it might cause a serious performance bottlenec
k
in the application.
*/
def
global
:
DataStream
[
T
]
=
javaStream
.
global
()
...
...
@@ -203,39 +203,78 @@ class DataStream[T](javaStream: JavaStream[T]) {
*
*/
def
max
(
position
:
Int
)
:
DataStream
[
T
]
=
aggregate
(
AggregationType
.
MAX
,
position
)
/**
* Applies an aggregation that that gives the current maximum of the data stream at
* the given field.
*
*/
def
max
(
field
:
String
)
:
DataStream
[
T
]
=
aggregate
(
AggregationType
.
MAX
,
field
)
/**
* Applies an aggregation that that gives the current minimum of the data stream at
* the given position.
*
*/
def
min
(
position
:
Int
)
:
DataStream
[
T
]
=
aggregate
(
AggregationType
.
MIN
,
position
)
/**
* Applies an aggregation that that gives the current minimum of the data stream at
* the given field.
*
*/
def
min
(
field
:
String
)
:
DataStream
[
T
]
=
aggregate
(
AggregationType
.
MIN
,
field
)
/**
* Applies an aggregation that sums the data stream at the given position.
*
*/
def
sum
(
position
:
Int
)
:
DataStream
[
T
]
=
aggregate
(
AggregationType
.
SUM
,
position
)
/**
* Applies an aggregation that sums the data stream at the given field.
*
*/
def
sum
(
field
:
String
)
:
DataStream
[
T
]
=
aggregate
(
AggregationType
.
SUM
,
field
)
/**
* Applies an aggregation that that gives the current minimum element of the data stream by
* the given position. When equality, the user can set to get the first or last element with
* the minimal value.
* the given position. When equality, the first element is returned with the minimal value.
*
*/
def
minBy
(
position
:
Int
)
:
DataStream
[
T
]
=
aggregate
(
AggregationType
.
MINBY
,
position
)
/**
* Applies an aggregation that that gives the current minimum element of the data stream by
* the given field. When equality, the first element is returned with the minimal value.
*
*/
def
minBy
(
position
:
Int
,
first
:
Boolean
=
true
)
:
DataStream
[
T
]
=
aggregate
(
AggregationType
.
MINBY
,
position
,
first
)
def
minBy
(
field
:
String
)
:
DataStream
[
T
]
=
aggregate
(
AggregationType
.
MINBY
,
field
)
/**
/**
* Applies an aggregation that that gives the current maximum element of the data stream by
* the given position. When equality, the first element is returned with the maximal value.
*
*/
def
maxBy
(
position
:
Int
)
:
DataStream
[
T
]
=
aggregate
(
AggregationType
.
MAXBY
,
position
)
/**
* Applies an aggregation that that gives the current maximum element of the data stream by
* the given position. When equality, the user can set to get the first or last element with
* the maximal value.
* the given field. When equality, the first element is returned with the maximal value.
*
*/
def
maxBy
(
position
:
Int
,
first
:
Boolean
=
true
)
:
DataStream
[
T
]
=
aggregate
(
AggregationType
.
MAXBY
,
position
,
first
)
def
maxBy
(
field
:
String
)
:
DataStream
[
T
]
=
aggregate
(
AggregationType
.
MAXBY
,
field
)
private
def
aggregate
(
aggregationType
:
AggregationType
,
field
:
String
)
:
DataStream
[
T
]
=
{
val
position
=
fieldNames2Indices
(
javaStream
.
getType
(),
Array
(
field
))(
0
)
aggregate
(
aggregationType
,
position
)
}
private
def
aggregate
(
aggregationType
:
AggregationType
,
position
:
Int
,
first
:
Boolean
=
true
)
:
private
def
aggregate
(
aggregationType
:
AggregationType
,
position
:
Int
)
:
DataStream
[
T
]
=
{
val
jStream
=
javaStream
.
asInstanceOf
[
JavaStream
[
Product
]]
...
...
@@ -246,7 +285,7 @@ class DataStream[T](javaStream: JavaStream[T]) {
val
reducer
=
aggregationType
match
{
case
AggregationType
.
SUM
=>
new
agg
.
Sum
(
SumFunction
.
getForClass
(
outType
.
getTypeAt
(
position
).
getTypeClass
()));
case
_
=>
new
agg
.
ProductComparableAggregator
(
aggregationType
,
first
)
case
_
=>
new
agg
.
ProductComparableAggregator
(
aggregationType
,
true
)
}
val
invokable
=
jStream
match
{
...
...
flink-addons/flink-streaming/flink-streaming-scala/src/main/scala/org/apache/flink/streaming/api/scala/WindowedDataStream.scala
浏览文件 @
06503c8f
...
...
@@ -157,6 +157,13 @@ class WindowedDataStream[T](javaStream: JavaWStream[T]) {
*
*/
def
max
(
position
:
Int
)
:
DataStream
[
T
]
=
aggregate
(
AggregationType
.
MAX
,
position
)
/**
* Applies an aggregation that that gives the maximum of the elements in the window at
* the given field.
*
*/
def
max
(
field
:
String
)
:
DataStream
[
T
]
=
aggregate
(
AggregationType
.
MAX
,
field
)
/**
* Applies an aggregation that that gives the minimum of the elements in the window at
...
...
@@ -164,30 +171,64 @@ class WindowedDataStream[T](javaStream: JavaWStream[T]) {
*
*/
def
min
(
position
:
Int
)
:
DataStream
[
T
]
=
aggregate
(
AggregationType
.
MIN
,
position
)
/**
* Applies an aggregation that that gives the minimum of the elements in the window at
* the given field.
*
*/
def
min
(
field
:
String
)
:
DataStream
[
T
]
=
aggregate
(
AggregationType
.
MIN
,
field
)
/**
* Applies an aggregation that sums the elements in the window at the given position.
*
*/
def
sum
(
position
:
Int
)
:
DataStream
[
T
]
=
aggregate
(
AggregationType
.
SUM
,
position
)
/**
* Applies an aggregation that sums the elements in the window at the given field.
*
*/
def
sum
(
field
:
String
)
:
DataStream
[
T
]
=
aggregate
(
AggregationType
.
SUM
,
field
)
/**
* Applies an aggregation that that gives the maximum element of the window by
* the given position. When equality, returns the first.
*
*/
def
maxBy
(
position
:
Int
,
first
:
Boolean
=
true
)
:
DataStream
[
T
]
=
aggregate
(
AggregationType
.
MAXBY
,
position
,
first
)
def
maxBy
(
position
:
Int
)
:
DataStream
[
T
]
=
aggregate
(
AggregationType
.
MAXBY
,
position
)
/**
* Applies an aggregation that that gives the maximum element of the window by
* the given field. When equality, returns the first.
*
*/
def
maxBy
(
field
:
String
)
:
DataStream
[
T
]
=
aggregate
(
AggregationType
.
MAXBY
,
field
)
/**
* Applies an aggregation that that gives the minimum element of the window by
* the given position. When equality, returns the first.
*
*/
def
minBy
(
position
:
Int
,
first
:
Boolean
=
true
)
:
DataStream
[
T
]
=
aggregate
(
AggregationType
.
MINBY
,
position
,
first
)
def
minBy
(
position
:
Int
)
:
DataStream
[
T
]
=
aggregate
(
AggregationType
.
MINBY
,
position
)
/**
* Applies an aggregation that that gives the minimum element of the window by
* the given field. When equality, returns the first.
*
*/
def
minBy
(
field
:
String
)
:
DataStream
[
T
]
=
aggregate
(
AggregationType
.
MINBY
,
field
)
private
def
aggregate
(
aggregationType
:
AggregationType
,
field
:
String
)
:
DataStream
[
T
]
=
{
val
position
=
fieldNames2Indices
(
javaStream
.
getType
(),
Array
(
field
))(
0
)
aggregate
(
aggregationType
,
position
)
}
def
aggregate
(
aggregationType
:
AggregationType
,
position
:
Int
,
first
:
Boolean
=
true
)
:
def
aggregate
(
aggregationType
:
AggregationType
,
position
:
Int
)
:
DataStream
[
T
]
=
{
val
jStream
=
javaStream
.
asInstanceOf
[
JavaWStream
[
Product
]]
...
...
@@ -198,7 +239,7 @@ class WindowedDataStream[T](javaStream: JavaWStream[T]) {
val
reducer
=
aggregationType
match
{
case
AggregationType
.
SUM
=>
new
agg
.
Sum
(
SumFunction
.
getForClass
(
outType
.
getTypeAt
(
position
).
getTypeClass
()));
case
_
=>
new
agg
.
ProductComparableAggregator
(
aggregationType
,
first
)
case
_
=>
new
agg
.
ProductComparableAggregator
(
aggregationType
,
true
)
}
new
DataStream
[
Product
](
jStream
.
reduce
(
reducer
)).
asInstanceOf
[
DataStream
[
T
]]
...
...
flink-addons/flink-streaming/flink-streaming-scala/src/main/scala/org/apache/flink/streaming/api/scala/package.scala
浏览文件 @
06503c8f
...
...
@@ -43,4 +43,24 @@ package object scala {
implicit
def
javaToScalaConnectedStream
[
IN1
,
IN2
](
javaStream
:
JavaConStream
[
IN1
,
IN2
])
:
ConnectedDataStream
[
IN1
,
IN2
]
=
new
ConnectedDataStream
[
IN1
,
IN2
](
javaStream
)
private
[
flink
]
def
fieldNames2Indices
(
typeInfo
:
TypeInformation
[
_
],
fields
:
Array
[
String
])
:
Array
[
Int
]
=
{
typeInfo
match
{
case
ti
:
CaseClassTypeInfo
[
_
]
=>
val
result
=
ti
.
getFieldIndices
(
fields
)
if
(
result
.
contains
(-
1
))
{
throw
new
IllegalArgumentException
(
"Fields '"
+
fields
.
mkString
(
", "
)
+
"' are not valid for '"
+
ti
.
toString
+
"'."
)
}
result
case
_
=>
throw
new
UnsupportedOperationException
(
"Specifying fields by name is only"
+
"supported on Case Classes (for now)."
)
}
}
}
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录