Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
Forever310
druid
提交
60b95dab
D
druid
项目概览
Forever310
/
druid
与 Fork 源项目一致
从无法访问的项目Fork
通知
3
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
DevOps
流水线
流水线任务
计划
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
D
druid
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
DevOps
DevOps
流水线
流水线任务
计划
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
流水线任务
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
60b95dab
编写于
4月 04, 2014
作者:
X
Xavier Léauté
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
CachePopulatingQueryRunner -> CachingQueryRunner
Populate and use cache at compute node level
上级
3e6ffbd5
变更
3
隐藏空白更改
内联
并排
Showing
3 changed file
with
135 addition
and
28 deletion
+135
-28
server/src/main/java/io/druid/client/CachingQueryRunner.java
server/src/main/java/io/druid/client/CachingQueryRunner.java
+60
-9
server/src/main/java/io/druid/server/coordination/ServerManager.java
...main/java/io/druid/server/coordination/ServerManager.java
+3
-3
server/src/test/java/io/druid/client/CachingQueryRunnerTest.java
...src/test/java/io/druid/client/CachingQueryRunnerTest.java
+72
-16
未找到文件。
server/src/main/java/io/druid/client/Cach
ePopulat
ingQueryRunner.java
→
server/src/main/java/io/druid/client/CachingQueryRunner.java
浏览文件 @
60b95dab
...
...
@@ -19,10 +19,14 @@
package
io.druid.client
;
import
com.fasterxml.jackson.core.type.TypeReference
;
import
com.fasterxml.jackson.databind.ObjectMapper
;
import
com.google.common.base.Function
;
import
com.google.common.base.Throwables
;
import
com.google.common.collect.Iterators
;
import
com.google.common.collect.Lists
;
import
com.google.common.util.concurrent.MoreExecutors
;
import
com.metamx.common.guava.BaseSequence
;
import
com.metamx.common.guava.Sequence
;
import
com.metamx.common.guava.Sequences
;
import
io.druid.client.cache.Cache
;
...
...
@@ -34,9 +38,11 @@ import io.druid.query.QueryToolChest;
import
io.druid.query.SegmentDescriptor
;
import
javax.annotation.Nullable
;
import
java.io.IOException
;
import
java.util.Iterator
;
import
java.util.List
;
public
class
Cach
ePopulat
ingQueryRunner
<
T
>
implements
QueryRunner
<
T
>
public
class
CachingQueryRunner
<
T
>
implements
QueryRunner
<
T
>
{
private
final
String
segmentIdentifier
;
...
...
@@ -47,9 +53,10 @@ public class CachePopulatingQueryRunner<T> implements QueryRunner<T>
private
final
ObjectMapper
mapper
;
private
final
CacheConfig
cacheConfig
;
public
Cach
ePopulat
ingQueryRunner
(
public
CachingQueryRunner
(
String
segmentIdentifier
,
SegmentDescriptor
segmentDescriptor
,
ObjectMapper
mapper
,
SegmentDescriptor
segmentDescriptor
,
ObjectMapper
mapper
,
Cache
cache
,
QueryToolChest
toolchest
,
QueryRunner
<
T
>
base
,
...
...
@@ -73,13 +80,57 @@ public class CachePopulatingQueryRunner<T> implements QueryRunner<T>
final
boolean
populateCache
=
query
.
getContextPopulateCache
(
true
)
&&
strategy
!=
null
&&
cacheConfig
.
isPopulateCache
();
if
(
populateCache
)
{
final
Cache
.
NamedKey
key
=
CacheUtil
.
computeSegmentCacheKey
(
segmentIdentifier
,
segmentDescriptor
,
strategy
.
computeCacheKey
(
query
)
);
final
boolean
useCache
=
Boolean
.
parseBoolean
(
query
.
getContextValue
(
CacheConfig
.
USE_CACHE
,
"true"
))
&&
strategy
!=
null
&&
cacheConfig
.
isPopulateCache
();
final
Cache
.
NamedKey
key
=
CacheUtil
.
computeSegmentCacheKey
(
segmentIdentifier
,
segmentDescriptor
,
strategy
.
computeCacheKey
(
query
)
);
if
(
useCache
)
{
final
Function
cacheFn
=
strategy
.
pullFromCache
();
final
byte
[]
cachedResult
=
cache
.
get
(
key
);
if
(
cachedResult
!=
null
)
{
final
TypeReference
cacheObjectClazz
=
strategy
.
getCacheObjectClazz
();
return
Sequences
.
map
(
new
BaseSequence
<>(
new
BaseSequence
.
IteratorMaker
<
T
,
Iterator
<
T
>>()
{
@Override
public
Iterator
<
T
>
make
()
{
try
{
if
(
cachedResult
.
length
==
0
)
{
return
Iterators
.
emptyIterator
();
}
return
mapper
.
readValues
(
mapper
.
getFactory
().
createParser
(
cachedResult
),
cacheObjectClazz
);
}
catch
(
IOException
e
)
{
throw
Throwables
.
propagate
(
e
);
}
}
@Override
public
void
cleanup
(
Iterator
<
T
>
iterFromMake
)
{
}
}
),
cacheFn
);
}
}
if
(
populateCache
)
{
final
Function
cacheFn
=
strategy
.
prepareForCache
();
final
List
<
Object
>
cacheResults
=
Lists
.
newLinkedList
();
...
...
server/src/main/java/io/druid/server/coordination/ServerManager.java
浏览文件 @
60b95dab
...
...
@@ -29,7 +29,7 @@ import com.metamx.common.guava.FunctionalIterable;
import
com.metamx.emitter.EmittingLogger
;
import
com.metamx.emitter.service.ServiceEmitter
;
import
com.metamx.emitter.service.ServiceMetricEvent
;
import
io.druid.client.Cach
ePopulat
ingQueryRunner
;
import
io.druid.client.CachingQueryRunner
;
import
io.druid.client.cache.Cache
;
import
io.druid.client.cache.CacheConfig
;
import
io.druid.collections.CountingMap
;
...
...
@@ -410,7 +410,7 @@ public class ServerManager implements QuerySegmentWalker
new
BySegmentQueryRunner
<
T
>(
adapter
.
getIdentifier
(),
adapter
.
getDataInterval
().
getStart
(),
new
Cach
ePopulat
ingQueryRunner
<
T
>(
new
CachingQueryRunner
<
T
>(
adapter
.
getIdentifier
(),
segmentDescriptor
,
objectMapper
,
...
...
@@ -424,4 +424,4 @@ public class ServerManager implements QuerySegmentWalker
segmentSpec
);
}
}
\ No newline at end of file
}
server/src/test/java/io/druid/client/Cach
ePopulat
ingQueryRunnerTest.java
→
server/src/test/java/io/druid/client/CachingQueryRunnerTest.java
浏览文件 @
60b95dab
...
...
@@ -19,9 +19,9 @@
package
io.druid.client
;
import
com.fasterxml.jackson.databind.ObjectMapper
;
import
com.google.common.base.Function
;
import
com.google.common.collect.ImmutableMap
;
import
com.google.common.collect.Iterables
;
import
com.google.common.collect.Iterators
;
import
com.google.common.collect.Lists
;
import
com.metamx.common.ISE
;
...
...
@@ -57,12 +57,11 @@ import java.io.Closeable;
import
java.io.IOException
;
import
java.util.ArrayList
;
import
java.util.Arrays
;
import
java.util.Iterator
;
import
java.util.List
;
import
java.util.Map
;
import
java.util.concurrent.atomic.AtomicBoolean
;
public
class
Cach
ePopulat
ingQueryRunnerTest
public
class
CachingQueryRunnerTest
{
private
static
final
List
<
AggregatorFactory
>
AGGS
=
Arrays
.
asList
(
...
...
@@ -71,17 +70,17 @@ public class CachePopulatingQueryRunnerTest
new
LongSumAggregatorFactory
(
"impers"
,
"imps"
)
);
private
static
final
Object
[]
objects
=
new
Object
[]{
new
DateTime
(
"2011-01-05"
),
"a"
,
50
,
4994
,
"b"
,
50
,
4993
,
"c"
,
50
,
4992
,
new
DateTime
(
"2011-01-06"
),
"a"
,
50
,
4991
,
"b"
,
50
,
4990
,
"c"
,
50
,
4989
,
new
DateTime
(
"2011-01-07"
),
"a"
,
50
,
4991
,
"b"
,
50
,
4990
,
"c"
,
50
,
4989
,
new
DateTime
(
"2011-01-08"
),
"a"
,
50
,
4988
,
"b"
,
50
,
4987
,
"c"
,
50
,
4986
,
new
DateTime
(
"2011-01-09"
),
"a"
,
50
,
4985
,
"b"
,
50
,
4984
,
"c"
,
50
,
4983
};
@Test
public
void
testC
achePopulatingQueryRunnerResourceClosing
()
throws
Exception
public
void
testC
loseAndPopulate
()
throws
Exception
{
Object
[]
objects
=
new
Object
[]{
new
DateTime
(
"2011-01-05"
),
"a"
,
50
,
4994
,
"b"
,
50
,
4993
,
"c"
,
50
,
4992
,
new
DateTime
(
"2011-01-06"
),
"a"
,
50
,
4991
,
"b"
,
50
,
4990
,
"c"
,
50
,
4989
,
new
DateTime
(
"2011-01-07"
),
"a"
,
50
,
4991
,
"b"
,
50
,
4990
,
"c"
,
50
,
4989
,
new
DateTime
(
"2011-01-08"
),
"a"
,
50
,
4988
,
"b"
,
50
,
4987
,
"c"
,
50
,
4986
,
new
DateTime
(
"2011-01-09"
),
"a"
,
50
,
4985
,
"b"
,
50
,
4984
,
"c"
,
50
,
4983
};
Iterable
<
Result
<
TopNResultValue
>>
expectedRes
=
makeTopNResults
(
false
,
objects
);
final
TopNQueryBuilder
builder
=
new
TopNQueryBuilder
()
.
dataSource
(
"ds"
)
...
...
@@ -114,10 +113,11 @@ public class CachePopulatingQueryRunnerTest
SegmentDescriptor
segmentDescriptor
=
new
SegmentDescriptor
(
new
Interval
(
"2011/2012"
),
"version"
,
0
);
TopNQueryQueryToolChest
toolchest
=
new
TopNQueryQueryToolChest
(
new
TopNQueryConfig
());
CachePopulatingQueryRunner
runner
=
new
CachePopulatingQueryRunner
(
DefaultObjectMapper
objectMapper
=
new
DefaultObjectMapper
();
CachingQueryRunner
runner
=
new
CachingQueryRunner
(
segmentIdentifier
,
segmentDescriptor
,
new
DefaultObjectMapper
()
,
objectMapper
,
cache
,
toolchest
,
new
QueryRunner
()
...
...
@@ -152,8 +152,6 @@ public class CachePopulatingQueryRunnerTest
Iterable
<
Result
<
TopNResultValue
>>
expectedCacheRes
=
makeTopNResults
(
true
,
objects
);
ObjectMapper
objectMapper
=
new
DefaultObjectMapper
();
byte
[]
cacheValue
=
cache
.
get
(
cacheKey
);
Assert
.
assertNotNull
(
cacheValue
);
...
...
@@ -170,6 +168,64 @@ public class CachePopulatingQueryRunnerTest
Assert
.
assertEquals
(
expectedCacheRes
,
cacheResults
);
}
@Test
public
void
testUseCache
()
throws
Exception
{
DefaultObjectMapper
objectMapper
=
new
DefaultObjectMapper
();
Iterable
<
Result
<
TopNResultValue
>>
expectedResults
=
makeTopNResults
(
true
,
objects
);
String
segmentIdentifier
=
"segment"
;
SegmentDescriptor
segmentDescriptor
=
new
SegmentDescriptor
(
new
Interval
(
"2011/2012"
),
"version"
,
0
);
TopNQueryQueryToolChest
toolchest
=
new
TopNQueryQueryToolChest
(
new
TopNQueryConfig
());
final
TopNQueryBuilder
builder
=
new
TopNQueryBuilder
()
.
dataSource
(
"ds"
)
.
dimension
(
"top_dim"
)
.
metric
(
"imps"
)
.
threshold
(
3
)
.
intervals
(
"2011-01-05/2011-01-10"
)
.
aggregators
(
AGGS
)
.
granularity
(
AllGranularity
.
ALL
);
final
TopNQuery
query
=
builder
.
build
();
CacheStrategy
<
Result
<
TopNResultValue
>,
Object
,
TopNQuery
>
cacheStrategy
=
toolchest
.
getCacheStrategy
(
query
);
Cache
.
NamedKey
cacheKey
=
CacheUtil
.
computeSegmentCacheKey
(
segmentIdentifier
,
segmentDescriptor
,
cacheStrategy
.
computeCacheKey
(
query
)
);
Cache
cache
=
MapCache
.
create
(
1024
*
1024
);
CacheUtil
.
populate
(
cache
,
objectMapper
,
cacheKey
,
Iterables
.
transform
(
expectedResults
,
cacheStrategy
.
prepareForCache
())
);
CachingQueryRunner
runner
=
new
CachingQueryRunner
(
segmentIdentifier
,
segmentDescriptor
,
objectMapper
,
cache
,
toolchest
,
// return an empty sequence since results should get pulled from cache
new
QueryRunner
()
{
@Override
public
Sequence
run
(
Query
query
)
{
return
Sequences
.
empty
();
}
},
new
CacheConfig
()
);
List
<
Object
>
results
=
Sequences
.
toList
(
runner
.
run
(
query
),
new
ArrayList
());
Assert
.
assertEquals
(
expectedResults
,
results
);
}
private
Iterable
<
Result
<
TopNResultValue
>>
makeTopNResults
(
boolean
cachedResults
,
Object
...
objects
)
{
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录