Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
Forever310
druid
提交
02f9cc3c
D
druid
项目概览
Forever310
/
druid
与 Fork 源项目一致
从无法访问的项目Fork
通知
3
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
DevOps
流水线
流水线任务
计划
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
D
druid
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
DevOps
DevOps
流水线
流水线任务
计划
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
流水线任务
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
02f9cc3c
编写于
8月 06, 2013
作者:
C
cheddar
浏览文件
操作
浏览文件
下载
差异文件
Merge pull request #212 from metamx/realtime-npe
Rejigger exception handling in standalone realtime
上级
930b089f
ec53a30c
变更
3
隐藏空白更改
内联
并排
Showing
3 changed file
with
46 addition
and
30 deletion
+46
-30
indexing-common/src/main/java/com/metamx/druid/indexer/data/InputRowParser.java
...in/java/com/metamx/druid/indexer/data/InputRowParser.java
+2
-1
indexing-common/src/main/java/com/metamx/druid/indexer/data/MapInputRowParser.java
...java/com/metamx/druid/indexer/data/MapInputRowParser.java
+20
-10
realtime/src/main/java/com/metamx/druid/realtime/RealtimeManager.java
.../main/java/com/metamx/druid/realtime/RealtimeManager.java
+24
-19
未找到文件。
indexing-common/src/main/java/com/metamx/druid/indexer/data/InputRowParser.java
浏览文件 @
02f9cc3c
package
com.metamx.druid.indexer.data
;
import
com.metamx.common.exception.FormattedException
;
import
com.metamx.druid.input.InputRow
;
public
interface
InputRowParser
<
T
>
{
public
InputRow
parse
(
T
input
);
public
InputRow
parse
(
T
input
)
throws
FormattedException
;
public
void
addDimensionExclusion
(
String
dimension
);
}
indexing-common/src/main/java/com/metamx/druid/indexer/data/MapInputRowParser.java
浏览文件 @
02f9cc3c
...
...
@@ -4,6 +4,7 @@ import com.fasterxml.jackson.annotation.JsonCreator;
import
com.fasterxml.jackson.annotation.JsonProperty
;
import
com.google.common.collect.Lists
;
import
com.google.common.collect.Sets
;
import
com.metamx.common.exception.FormattedException
;
import
com.metamx.druid.input.InputRow
;
import
com.metamx.druid.input.MapBasedInputRow
;
import
org.joda.time.DateTime
;
...
...
@@ -37,21 +38,30 @@ public class MapInputRowParser implements InputRowParser<Map<String, Object>>
}
@Override
public
InputRow
parse
(
Map
<
String
,
Object
>
theMap
)
public
InputRow
parse
(
Map
<
String
,
Object
>
theMap
)
throws
FormattedException
{
final
List
<
String
>
dimensions
=
dataSpec
.
hasCustomDimensions
()
?
dataSpec
.
getDimensions
()
:
Lists
.
newArrayList
(
Sets
.
difference
(
theMap
.
keySet
(),
dimensionExclusions
));
final
DateTime
timestamp
=
timestampSpec
.
extractTimestamp
(
theMap
);
if
(
timestamp
==
null
)
{
final
String
input
=
theMap
.
toString
();
throw
new
NullPointerException
(
String
.
format
(
"Null timestamp in input: %s"
,
input
.
length
()
<
100
?
input
:
input
.
substring
(
0
,
100
)
+
"..."
)
);
final
DateTime
timestamp
;
try
{
timestamp
=
timestampSpec
.
extractTimestamp
(
theMap
);
if
(
timestamp
==
null
)
{
final
String
input
=
theMap
.
toString
();
throw
new
NullPointerException
(
String
.
format
(
"Null timestamp in input: %s"
,
input
.
length
()
<
100
?
input
:
input
.
substring
(
0
,
100
)
+
"..."
)
);
}
}
catch
(
Exception
e
)
{
throw
new
FormattedException
.
Builder
()
.
withErrorCode
(
FormattedException
.
ErrorCode
.
UNPARSABLE_TIMESTAMP
)
.
withMessage
(
e
.
toString
())
.
build
();
}
return
new
MapBasedInputRow
(
timestamp
.
getMillis
(),
dimensions
,
theMap
);
...
...
realtime/src/main/java/com/metamx/druid/realtime/RealtimeManager.java
浏览文件 @
02f9cc3c
...
...
@@ -164,32 +164,37 @@ public class RealtimeManager implements QuerySegmentWalker
final
InputRow
inputRow
;
try
{
inputRow
=
firehose
.
nextRow
();
}
catch
(
FormattedException
e
)
{
log
.
info
(
e
,
"unparseable line: %s"
,
e
.
getDetails
());
metrics
.
incrementUnparseable
();
continue
;
}
catch
(
Exception
e
)
{
log
.
info
(
e
,
"thrown away line due to exception"
);
metrics
.
incrementThrownAway
();
continue
;
}
final
Sink
sink
=
plumber
.
getSink
(
inputRow
.
getTimestampFromEpoch
());
if
(
sink
==
null
)
{
metrics
.
incrementThrownAway
();
log
.
debug
(
"Throwing away event[%s]"
,
inputRow
);
if
(
System
.
currentTimeMillis
()
>
nextFlush
)
{
plumber
.
persist
(
firehose
.
commit
());
nextFlush
=
new
DateTime
().
plus
(
intermediatePersistPeriod
).
getMillis
();
}
continue
;
}
final
Sink
sink
=
plumber
.
getSink
(
inputRow
.
getTimestampFromEpoch
());
if
(
sink
==
null
)
{
metrics
.
incrementThrownAway
();
log
.
debug
(
"Throwing away event[%s]"
,
inputRow
);
int
currCount
=
sink
.
add
(
inputRow
);
metrics
.
incrementProcessed
();
if
(
currCount
>=
config
.
getMaxRowsInMemory
()
||
System
.
currentTimeMillis
()
>
nextFlush
)
{
if
(
System
.
currentTimeMillis
()
>
nextFlush
)
{
plumber
.
persist
(
firehose
.
commit
());
nextFlush
=
new
DateTime
().
plus
(
intermediatePersistPeriod
).
getMillis
();
}
}
catch
(
FormattedException
e
)
{
log
.
info
(
e
,
"unparseable line: %s"
,
e
.
getDetails
());
metrics
.
incrementUnparseable
();
continue
;
}
int
currCount
=
sink
.
add
(
inputRow
);
metrics
.
incrementProcessed
();
if
(
currCount
>=
config
.
getMaxRowsInMemory
()
||
System
.
currentTimeMillis
()
>
nextFlush
)
{
plumber
.
persist
(
firehose
.
commit
());
nextFlush
=
new
DateTime
().
plus
(
intermediatePersistPeriod
).
getMillis
();
}
}
}
catch
(
RuntimeException
e
)
{
log
.
makeAlert
(
e
,
"RuntimeException aborted realtime processing[%s]"
,
fireDepartment
.
getSchema
().
getDataSource
())
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录