Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
Forever310
druid
提交
a1a5e425
D
druid
项目概览
Forever310
/
druid
与 Fork 源项目一致
从无法访问的项目Fork
通知
3
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
DevOps
流水线
流水线任务
计划
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
D
druid
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
DevOps
DevOps
流水线
流水线任务
计划
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
流水线任务
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
a1a5e425
编写于
7月 08, 2014
作者:
N
nishantmonu51
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
CombiningFirehoseFactory
can be used to ingest data from multiple sources.
上级
97b58eb1
变更
3
隐藏空白更改
内联
并排
Showing
3 changed file
with
275 addition
and
1 deletion
+275
-1
server/src/main/java/io/druid/guice/FirehoseModule.java
server/src/main/java/io/druid/guice/FirehoseModule.java
+3
-1
server/src/main/java/io/druid/segment/realtime/firehose/CombiningFirehoseFactory.java
...d/segment/realtime/firehose/CombiningFirehoseFactory.java
+125
-0
server/src/test/java/io/druid/realtime/firehose/CombiningFirehoseFactoryTest.java
...druid/realtime/firehose/CombiningFirehoseFactoryTest.java
+147
-0
未找到文件。
server/src/main/java/io/druid/guice/FirehoseModule.java
浏览文件 @
a1a5e425
...
@@ -26,6 +26,7 @@ import com.google.inject.Binder;
...
@@ -26,6 +26,7 @@ import com.google.inject.Binder;
import
io.druid.data.input.ProtoBufInputRowParser
;
import
io.druid.data.input.ProtoBufInputRowParser
;
import
io.druid.initialization.DruidModule
;
import
io.druid.initialization.DruidModule
;
import
io.druid.segment.realtime.firehose.ClippedFirehoseFactory
;
import
io.druid.segment.realtime.firehose.ClippedFirehoseFactory
;
import
io.druid.segment.realtime.firehose.CombiningFirehoseFactory
;
import
io.druid.segment.realtime.firehose.EventReceiverFirehoseFactory
;
import
io.druid.segment.realtime.firehose.EventReceiverFirehoseFactory
;
import
io.druid.segment.realtime.firehose.IrcFirehoseFactory
;
import
io.druid.segment.realtime.firehose.IrcFirehoseFactory
;
import
io.druid.segment.realtime.firehose.LocalFirehoseFactory
;
import
io.druid.segment.realtime.firehose.LocalFirehoseFactory
;
...
@@ -53,7 +54,8 @@ public class FirehoseModule implements DruidModule
...
@@ -53,7 +54,8 @@ public class FirehoseModule implements DruidModule
new
NamedType
(
TimedShutoffFirehoseFactory
.
class
,
"timed"
),
new
NamedType
(
TimedShutoffFirehoseFactory
.
class
,
"timed"
),
new
NamedType
(
IrcFirehoseFactory
.
class
,
"irc"
),
new
NamedType
(
IrcFirehoseFactory
.
class
,
"irc"
),
new
NamedType
(
LocalFirehoseFactory
.
class
,
"local"
),
new
NamedType
(
LocalFirehoseFactory
.
class
,
"local"
),
new
NamedType
(
EventReceiverFirehoseFactory
.
class
,
"receiver"
)
new
NamedType
(
EventReceiverFirehoseFactory
.
class
,
"receiver"
),
new
NamedType
(
CombiningFirehoseFactory
.
class
,
"combining"
)
)
)
);
);
}
}
...
...
server/src/main/java/io/druid/segment/realtime/firehose/CombiningFirehoseFactory.java
0 → 100644
浏览文件 @
a1a5e425
/*
* Druid - a distributed column store.
* Copyright (C) 2012, 2013 Metamarkets Group Inc.
*
* This program is free software; you can redistribute it and/or
* modify it under the terms of the GNU General Public License
* as published by the Free Software Foundation; either version 2
* of the License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program; if not, write to the Free Software
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
*/
package
io.druid.segment.realtime.firehose
;
import
com.fasterxml.jackson.annotation.JsonCreator
;
import
com.fasterxml.jackson.annotation.JsonProperty
;
import
com.google.api.client.repackaged.com.google.common.base.Preconditions
;
import
com.google.api.client.repackaged.com.google.common.base.Throwables
;
import
io.druid.data.input.Firehose
;
import
io.druid.data.input.FirehoseFactory
;
import
io.druid.data.input.InputRow
;
import
io.druid.data.input.impl.InputRowParser
;
import
java.io.IOException
;
import
java.util.Iterator
;
import
java.util.List
;
/**
* Creates firehose that combines data from different Firehoses. Useful for ingesting data from multiple sources.
*/
public
class
CombiningFirehoseFactory
implements
FirehoseFactory
<
InputRowParser
>
{
private
final
List
<
FirehoseFactory
>
delegateFactoryList
;
@JsonCreator
public
CombiningFirehoseFactory
(
@JsonProperty
(
"delegates"
)
List
<
FirehoseFactory
>
delegateFactoryList
)
{
Preconditions
.
checkArgument
(!
delegateFactoryList
.
isEmpty
());
this
.
delegateFactoryList
=
delegateFactoryList
;
}
@Override
public
Firehose
connect
(
InputRowParser
parser
)
throws
IOException
{
return
new
CombiningFirehose
(
parser
);
}
@Override
public
InputRowParser
getParser
()
{
return
delegateFactoryList
.
get
(
0
).
getParser
();
}
@JsonProperty
(
"delegates"
)
public
List
<
FirehoseFactory
>
getDelegateFactoryList
()
{
return
delegateFactoryList
;
}
public
class
CombiningFirehose
implements
Firehose
{
private
final
InputRowParser
parser
;
private
final
Iterator
<
FirehoseFactory
>
firehoseFactoryIterator
;
private
volatile
Firehose
currentFirehose
;
public
CombiningFirehose
(
InputRowParser
parser
)
throws
IOException
{
this
.
firehoseFactoryIterator
=
delegateFactoryList
.
iterator
();
this
.
parser
=
parser
;
nextFirehose
();
}
private
void
nextFirehose
()
{
if
(
firehoseFactoryIterator
.
hasNext
())
{
try
{
if
(
currentFirehose
!=
null
)
{
currentFirehose
.
close
();
}
currentFirehose
=
firehoseFactoryIterator
.
next
().
connect
(
parser
);
}
catch
(
IOException
e
)
{
Throwables
.
propagate
(
e
);
}
}
}
@Override
public
boolean
hasMore
()
{
return
currentFirehose
.
hasMore
();
}
@Override
public
InputRow
nextRow
()
{
InputRow
rv
=
currentFirehose
.
nextRow
();
if
(!
currentFirehose
.
hasMore
())
{
nextFirehose
();
}
return
rv
;
}
@Override
public
Runnable
commit
()
{
return
currentFirehose
.
commit
();
}
@Override
public
void
close
()
throws
IOException
{
currentFirehose
.
close
();
}
}
}
server/src/test/java/io/druid/realtime/firehose/CombiningFirehoseFactoryTest.java
0 → 100644
浏览文件 @
a1a5e425
/*
* Druid - a distributed column store.
* Copyright (C) 2012, 2013, 2014 Metamarkets Group Inc.
*
* This program is free software; you can redistribute it and/or
* modify it under the terms of the GNU General Public License
* as published by the Free Software Foundation; either version 2
* of the License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program; if not, write to the Free Software
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
*/
package
io.druid.realtime.firehose
;
import
com.google.common.collect.Lists
;
import
com.metamx.common.parsers.ParseException
;
import
io.druid.data.input.Firehose
;
import
io.druid.data.input.FirehoseFactory
;
import
io.druid.data.input.InputRow
;
import
io.druid.data.input.impl.InputRowParser
;
import
io.druid.segment.realtime.firehose.CombiningFirehoseFactory
;
import
io.druid.utils.Runnables
;
import
org.junit.Assert
;
import
org.junit.Test
;
import
java.io.IOException
;
import
java.util.Arrays
;
import
java.util.Iterator
;
import
java.util.List
;
public
class
CombiningFirehoseFactoryTest
{
@Test
public
void
testCombiningfirehose
()
throws
IOException
{
List
<
InputRow
>
list1
=
Arrays
.
asList
(
makeRow
(
1
,
1
),
makeRow
(
2
,
2
));
List
<
InputRow
>
list2
=
Arrays
.
asList
(
makeRow
(
3
,
3
),
makeRow
(
4
,
4
));
FirehoseFactory
combiningFactory
=
new
CombiningFirehoseFactory
(
Arrays
.<
FirehoseFactory
>
asList
(
new
ListFirehoseFactory
(
list1
),
new
ListFirehoseFactory
(
list2
)
)
);
final
Firehose
firehose
=
combiningFactory
.
connect
(
null
);
for
(
int
i
=
1
;
i
<
5
;
i
++)
{
Assert
.
assertTrue
(
firehose
.
hasMore
());
final
InputRow
inputRow
=
firehose
.
nextRow
();
Assert
.
assertEquals
(
i
,
inputRow
.
getTimestampFromEpoch
());
Assert
.
assertEquals
(
i
,
inputRow
.
getFloatMetric
(
"test"
),
0
);
}
Assert
.
assertFalse
(
firehose
.
hasMore
());
}
private
InputRow
makeRow
(
final
long
timestamp
,
final
float
metricValue
)
{
return
new
InputRow
()
{
@Override
public
List
<
String
>
getDimensions
()
{
return
Arrays
.
asList
(
"testDim"
);
}
@Override
public
long
getTimestampFromEpoch
()
{
return
timestamp
;
}
@Override
public
List
<
String
>
getDimension
(
String
dimension
)
{
return
Lists
.
newArrayList
();
}
@Override
public
float
getFloatMetric
(
String
metric
)
{
return
metricValue
;
}
@Override
public
Object
getRaw
(
String
dimension
)
{
return
null
;
}
};
}
public
static
class
ListFirehoseFactory
implements
FirehoseFactory
<
InputRowParser
>
{
private
final
List
<
InputRow
>
rows
;
ListFirehoseFactory
(
List
<
InputRow
>
rows
)
{
this
.
rows
=
rows
;
}
@Override
public
Firehose
connect
(
InputRowParser
inputRowParser
)
throws
IOException
,
ParseException
{
final
Iterator
<
InputRow
>
iterator
=
rows
.
iterator
();
return
new
Firehose
()
{
@Override
public
boolean
hasMore
()
{
return
iterator
.
hasNext
();
}
@Override
public
InputRow
nextRow
()
{
return
iterator
.
next
();
}
@Override
public
Runnable
commit
()
{
return
Runnables
.
getNoopRunnable
();
}
@Override
public
void
close
()
throws
IOException
{
//
}
};
}
@Override
public
InputRowParser
getParser
()
{
return
null
;
}
}
}
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录