Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
Forever310
druid
提交
4c133272
D
druid
项目概览
Forever310
/
druid
与 Fork 源项目一致
从无法访问的项目Fork
通知
3
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
DevOps
流水线
流水线任务
计划
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
D
druid
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
DevOps
DevOps
流水线
流水线任务
计划
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
流水线任务
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
4c133272
编写于
5月 30, 2014
作者:
F
fjy
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
more logging for determine hashed
上级
ee60183d
变更
3
隐藏空白更改
内联
并排
Showing
3 changed file
with
30 addition
and
14 deletion
+30
-14
indexing-hadoop/src/main/java/io/druid/indexer/DetermineHashedPartitionsJob.java
...n/java/io/druid/indexer/DetermineHashedPartitionsJob.java
+19
-14
indexing-hadoop/src/main/java/io/druid/indexer/HadoopDruidIndexerConfig.java
.../main/java/io/druid/indexer/HadoopDruidIndexerConfig.java
+1
-0
indexing-service/src/main/java/io/druid/indexing/common/task/IndexTask.java
...rc/main/java/io/druid/indexing/common/task/IndexTask.java
+10
-0
未找到文件。
indexing-hadoop/src/main/java/io/druid/indexer/DetermineHashedPartitionsJob.java
浏览文件 @
4c133272
...
...
@@ -158,22 +158,27 @@ public class DetermineHashedPartitionsJob implements Jobby
fileSystem
=
partitionInfoPath
.
getFileSystem
(
groupByJob
.
getConfiguration
());
}
if
(
Utils
.
exists
(
groupByJob
,
fileSystem
,
partitionInfoPath
))
{
Long
cardinality
=
config
.
jsonMapper
.
readValue
(
Utils
.
openInputStream
(
groupByJob
,
partitionInfoPath
),
new
TypeReference
<
Long
>()
{
}
);
final
int
numberOfShards
=
(
int
)
Math
.
ceil
((
double
)
cardinality
/
config
.
getTargetPartitionSize
());
final
Long
numRows
=
config
.
jsonMapper
.
readValue
(
Utils
.
openInputStream
(
groupByJob
,
partitionInfoPath
),
new
TypeReference
<
Long
>()
{
}
);
List
<
HadoopyShardSpec
>
actualSpecs
=
Lists
.
newArrayListWithExpectedSize
(
numberOfShards
);
if
(
numberOfShards
==
1
)
{
actualSpecs
.
add
(
new
HadoopyShardSpec
(
new
NoneShardSpec
(),
shardCount
++));
}
else
{
for
(
int
i
=
0
;
i
<
numberOfShards
;
++
i
)
{
actualSpecs
.
add
(
new
HadoopyShardSpec
(
new
HashBasedNumberedShardSpec
(
i
,
numberOfShards
),
shardCount
++));
log
.
info
(
"DateTime[%s], partition[%d], spec[%s]"
,
bucket
,
i
,
actualSpecs
.
get
(
i
));
log
.
info
(
"Found approximately [%,d] rows in data."
,
numRows
);
final
int
numberOfShards
=
(
int
)
Math
.
ceil
((
double
)
numRows
/
config
.
getTargetPartitionSize
());
log
.
info
(
"Creating [%,d] shards"
,
numberOfShards
);
List
<
HadoopyShardSpec
>
actualSpecs
=
Lists
.
newArrayListWithExpectedSize
(
numberOfShards
);
if
(
numberOfShards
==
1
)
{
actualSpecs
.
add
(
new
HadoopyShardSpec
(
new
NoneShardSpec
(),
shardCount
++));
}
else
{
for
(
int
i
=
0
;
i
<
numberOfShards
;
++
i
)
{
actualSpecs
.
add
(
new
HadoopyShardSpec
(
new
HashBasedNumberedShardSpec
(
i
,
numberOfShards
),
shardCount
++));
log
.
info
(
"DateTime[%s], partition[%d], spec[%s]"
,
bucket
,
i
,
actualSpecs
.
get
(
i
));
}
}
}
shardSpecs
.
put
(
bucket
,
actualSpecs
);
...
...
indexing-hadoop/src/main/java/io/druid/indexer/HadoopDruidIndexerConfig.java
浏览文件 @
4c133272
...
...
@@ -37,6 +37,7 @@ import com.google.inject.Module;
import
com.metamx.common.ISE
;
import
com.metamx.common.guava.FunctionalIterable
;
import
com.metamx.common.logger.Logger
;
import
com.metamx.common.parsers.TimestampParser
;
import
io.druid.common.utils.JodaUtils
;
import
io.druid.data.input.InputRow
;
import
io.druid.data.input.impl.StringInputRowParser
;
...
...
indexing-service/src/main/java/io/druid/indexing/common/task/IndexTask.java
浏览文件 @
4c133272
...
...
@@ -23,6 +23,7 @@ import com.fasterxml.jackson.annotation.JsonCreator;
import
com.fasterxml.jackson.annotation.JsonIgnore
;
import
com.fasterxml.jackson.annotation.JsonProperty
;
import
com.fasterxml.jackson.annotation.JsonTypeName
;
import
com.google.common.base.Function
;
import
com.google.common.collect.ImmutableList
;
import
com.google.common.collect.Iterables
;
import
com.google.common.collect.Lists
;
...
...
@@ -34,6 +35,7 @@ import com.google.common.primitives.Ints;
import
com.metamx.common.ISE
;
import
com.metamx.common.guava.Comparators
;
import
com.metamx.common.logger.Logger
;
import
com.metamx.common.parsers.TimestampParser
;
import
io.druid.data.input.Firehose
;
import
io.druid.data.input.FirehoseFactory
;
import
io.druid.data.input.InputRow
;
...
...
@@ -551,4 +553,12 @@ public class IndexTask extends AbstractFixedIntervalTask
return
rowFlushBoundary
;
}
}
public
static
void
main
(
String
[]
args
)
{
Function
<
String
,
DateTime
>
parser
=
TimestampParser
.
createTimestampParser
(
"millis"
);
parser
.
apply
(
"1401266370985"
);
}
}
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录