Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
Forever310
druid
提交
2580a9c7
D
druid
项目概览
Forever310
/
druid
与 Fork 源项目一致
从无法访问的项目Fork
通知
3
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
DevOps
流水线
流水线任务
计划
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
D
druid
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
DevOps
DevOps
流水线
流水线任务
计划
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
流水线任务
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
2580a9c7
编写于
3月 26, 2014
作者:
X
Xavier Léauté
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
mark long-running tests as @Ignore + docs
上级
f42b850d
变更
1
隐藏空白更改
内联
并排
Showing
1 changed file
with
18 addition
and
5 deletion
+18
-5
processing/src/test/java/io/druid/query/aggregation/hyperloglog/HyperLogLogCollectorTest.java
...ery/aggregation/hyperloglog/HyperLogLogCollectorTest.java
+18
-5
未找到文件。
processing/src/test/java/io/druid/query/aggregation/hyperloglog/HyperLogLogCollectorTest.java
浏览文件 @
2580a9c7
...
@@ -22,6 +22,7 @@ package io.druid.query.aggregation.hyperloglog;
...
@@ -22,6 +22,7 @@ package io.druid.query.aggregation.hyperloglog;
import
com.google.common.hash.HashFunction
;
import
com.google.common.hash.HashFunction
;
import
com.google.common.hash.Hashing
;
import
com.google.common.hash.Hashing
;
import
org.junit.Assert
;
import
org.junit.Assert
;
import
org.junit.Ignore
;
import
org.junit.Test
;
import
org.junit.Test
;
import
java.nio.ByteBuffer
;
import
java.nio.ByteBuffer
;
...
@@ -70,17 +71,29 @@ public class HyperLogLogCollectorTest
...
@@ -70,17 +71,29 @@ public class HyperLogLogCollectorTest
}
}
}
}
// @Test
/**
* This is a very long running test, disabled by default.
* It is meant to catch issues when combining a large numer of HLL objects.
*
* It compares adding all the values to one HLL vs.
* splitting up values into HLLs of 100 values each, and folding those HLLs into a single main HLL.
*
* When reaching very large cardinalities (>> 50,000,000), offsets are mismatched between the main HLL and the ones
* with 100 values, requiring a floating max as described in
* http://druid.io/blog/2014/02/18/hyperloglog-optimizations-for-real-world-systems.html
*/
@Ignore
@Test
public
void
testHighCardinalityRollingFold
()
throws
Exception
public
void
testHighCardinalityRollingFold
()
throws
Exception
{
{
final
HyperLogLogCollector
rolling
=
HyperLogLogCollector
.
makeLatestCollector
();
final
HyperLogLogCollector
rolling
=
HyperLogLogCollector
.
makeLatestCollector
();
final
HyperLogLogCollector
simple
=
HyperLogLogCollector
.
makeLatestCollector
();
final
HyperLogLogCollector
simple
=
HyperLogLogCollector
.
makeLatestCollector
();
int
count
;
MessageDigest
md
=
MessageDigest
.
getInstance
(
"SHA-1"
);
MessageDigest
md
=
MessageDigest
.
getInstance
(
"SHA-1"
);
HyperLogLogCollector
tmp
=
HyperLogLogCollector
.
makeLatestCollector
();
HyperLogLogCollector
tmp
=
HyperLogLogCollector
.
makeLatestCollector
();
for
(
count
=
0
;
count
<
5000000
;
++
count
)
{
int
count
;
for
(
count
=
0
;
count
<
100_000_000
;
++
count
)
{
md
.
update
(
Integer
.
toString
(
count
).
getBytes
());
md
.
update
(
Integer
.
toString
(
count
).
getBytes
());
byte
[]
hashed
=
fn
.
hashBytes
(
md
.
digest
()).
asBytes
();
byte
[]
hashed
=
fn
.
hashBytes
(
md
.
digest
()).
asBytes
();
...
@@ -110,14 +123,14 @@ public class HyperLogLogCollectorTest
...
@@ -110,14 +123,14 @@ public class HyperLogLogCollectorTest
Assert
.
assertEquals
(
n
,
rolling
.
estimateCardinality
(),
n
*
0.05
);
Assert
.
assertEquals
(
n
,
rolling
.
estimateCardinality
(),
n
*
0.05
);
}
}
//
@Test
@Ignore
@Test
public
void
testHighCardinalityRollingFold2
()
throws
Exception
public
void
testHighCardinalityRollingFold2
()
throws
Exception
{
{
final
HyperLogLogCollector
rolling
=
HyperLogLogCollector
.
makeLatestCollector
();
final
HyperLogLogCollector
rolling
=
HyperLogLogCollector
.
makeLatestCollector
();
int
count
;
int
count
;
long
start
=
System
.
currentTimeMillis
();
long
start
=
System
.
currentTimeMillis
();
for
(
count
=
0
;
count
<
50
00
000
;
++
count
)
{
for
(
count
=
0
;
count
<
50
_000_
000
;
++
count
)
{
HyperLogLogCollector
theCollector
=
HyperLogLogCollector
.
makeLatestCollector
();
HyperLogLogCollector
theCollector
=
HyperLogLogCollector
.
makeLatestCollector
();
theCollector
.
add
(
fn
.
hashLong
(
count
).
asBytes
());
theCollector
.
add
(
fn
.
hashLong
(
count
).
asBytes
());
rolling
.
fold
(
theCollector
);
rolling
.
fold
(
theCollector
);
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录