Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
Forever310
druid
提交
7547e989
D
druid
项目概览
Forever310
/
druid
与 Fork 源项目一致
从无法访问的项目Fork
通知
3
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
DevOps
流水线
流水线任务
计划
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
D
druid
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
DevOps
DevOps
流水线
流水线任务
计划
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
流水线任务
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
7547e989
编写于
4月 02, 2014
作者:
F
fjy
浏览文件
操作
浏览文件
下载
差异文件
Merge pull request #456 from metamx/processing-threads-default
more sensible defaults
上级
5fae4d9a
2eb94930
变更
4
隐藏空白更改
内联
并排
Showing
4 changed file
with
78 addition
and
32 deletion
+78
-32
server/src/main/java/io/druid/guice/DruidProcessingModule.java
...r/src/main/java/io/druid/guice/DruidProcessingModule.java
+18
-31
server/src/main/java/io/druid/server/DruidProcessingConfig.java
.../src/main/java/io/druid/server/DruidProcessingConfig.java
+8
-0
server/src/main/java/io/druid/server/VMUtils.java
server/src/main/java/io/druid/server/VMUtils.java
+51
-0
services/src/main/java/io/druid/cli/CliRealtimeExample.java
services/src/main/java/io/druid/cli/CliRealtimeExample.java
+1
-1
未找到文件。
server/src/main/java/io/druid/guice/DruidProcessingModule.java
浏览文件 @
7547e989
...
...
@@ -31,17 +31,15 @@ import com.metamx.common.logger.Logger;
import
com.metamx.emitter.service.ServiceEmitter
;
import
com.metamx.emitter.service.ServiceMetricEvent
;
import
io.druid.collections.StupidPool
;
import
io.druid.concurrent.Execs
;
import
io.druid.guice.annotations.Global
;
import
io.druid.guice.annotations.Processing
;
import
io.druid.query.MetricsEmittingExecutorService
;
import
io.druid.query.PrioritizedExecutorService
;
import
io.druid.server.DruidProcessingConfig
;
import
io.druid.server.VMUtils
;
import
java.lang.reflect.InvocationTargetException
;
import
java.nio.ByteBuffer
;
import
java.util.concurrent.ExecutorService
;
import
java.util.concurrent.Executors
;
import
java.util.concurrent.atomic.AtomicLong
;
/**
...
...
@@ -82,37 +80,26 @@ public class DruidProcessingModule implements Module
public
StupidPool
<
ByteBuffer
>
getIntermediateResultsPool
(
DruidProcessingConfig
config
)
{
try
{
Class
<?>
vmClass
=
Class
.
forName
(
"sun.misc.VM"
);
Object
maxDirectMemoryObj
=
vmClass
.
getMethod
(
"maxDirectMemory"
).
invoke
(
null
);
long
maxDirectMemory
=
VMUtils
.
getMaxDirectMemory
();
if
(
maxDirectMemoryObj
==
null
||
!(
maxDirectMemoryObj
instanceof
Number
))
{
log
.
info
(
"Cannot determine maxDirectMemory from[%s]"
,
maxDirectMemoryObj
);
}
else
{
long
maxDirectMemory
=
((
Number
)
maxDirectMemoryObj
).
longValue
();
final
long
memoryNeeded
=
(
long
)
config
.
intermediateComputeSizeBytes
()
*
(
config
.
getNumThreads
()
+
1
);
if
(
maxDirectMemory
<
memoryNeeded
)
{
throw
new
ProvisionException
(
String
.
format
(
"Not enough direct memory. Please adjust -XX:MaxDirectMemorySize or druid.computation.buffer.size: "
+
"maxDirectMemory[%,d], memoryNeeded[%,d], druid.computation.buffer.size[%,d], numThreads[%,d]"
,
maxDirectMemory
,
memoryNeeded
,
config
.
intermediateComputeSizeBytes
(),
config
.
getNumThreads
()
)
);
}
final
long
memoryNeeded
=
(
long
)
config
.
intermediateComputeSizeBytes
()
*
(
config
.
getNumThreads
()
+
1
);
if
(
maxDirectMemory
<
memoryNeeded
)
{
throw
new
ProvisionException
(
String
.
format
(
"Not enough direct memory. Please adjust -XX:MaxDirectMemorySize, druid.computation.buffer.size, or druid.processing.numThreads: "
+
"maxDirectMemory[%,d], memoryNeeded[%,d] = druid.computation.buffer.size[%,d] * ( druid.processing.numThreads[%,d] + 1 )"
,
maxDirectMemory
,
memoryNeeded
,
config
.
intermediateComputeSizeBytes
(),
config
.
getNumThreads
()
)
);
}
}
catch
(
UnsupportedOperationException
e
)
{
log
.
info
(
e
.
getMessage
());
}
catch
(
ClassNotFoundException
e
)
{
log
.
info
(
"No VM class, cannot do memory check."
);
}
catch
(
NoSuchMethodException
e
)
{
log
.
info
(
"VM.maxDirectMemory doesn't exist, cannot do memory check."
);
}
catch
(
InvocationTargetException
e
)
{
log
.
warn
(
e
,
"static method shouldn't throw this"
);
}
catch
(
IllegalAccessException
e
)
{
log
.
warn
(
e
,
"public method, shouldn't throw this"
);
catch
(
RuntimeException
e
)
{
log
.
warn
(
e
,
e
.
getMessage
());
}
return
new
IntermediateProcessingBufferPool
(
config
.
intermediateComputeSizeBytes
());
...
...
server/src/main/java/io/druid/server/DruidProcessingConfig.java
浏览文件 @
7547e989
...
...
@@ -31,4 +31,12 @@ public abstract class DruidProcessingConfig extends ExecutorServiceConfig
{
return
1024
*
1024
*
1024
;
}
@Override
@Config
(
value
=
"${base_path}.numThreads"
)
public
int
getNumThreads
()
{
// default to leaving one core for background tasks
final
int
processors
=
Runtime
.
getRuntime
().
availableProcessors
();
return
processors
>
1
?
processors
-
1
:
processors
;
}
}
server/src/main/java/io/druid/server/VMUtils.java
0 → 100644
浏览文件 @
7547e989
/*
* Druid - a distributed column store.
* Copyright (C) 2012, 2013, 2014 Metamarkets Group Inc.
*
* This program is free software; you can redistribute it and/or
* modify it under the terms of the GNU General Public License
* as published by the Free Software Foundation; either version 2
* of the License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program; if not, write to the Free Software
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
*/
package
io.druid.server
;
import
java.lang.reflect.InvocationTargetException
;
public
class
VMUtils
{
public
static
long
getMaxDirectMemory
()
throws
UnsupportedOperationException
{
try
{
Class
<?>
vmClass
=
Class
.
forName
(
"sun.misc.VM"
);
Object
maxDirectMemoryObj
=
vmClass
.
getMethod
(
"maxDirectMemory"
).
invoke
(
null
);
if
(
maxDirectMemoryObj
==
null
||
!(
maxDirectMemoryObj
instanceof
Number
))
{
throw
new
UnsupportedOperationException
(
String
.
format
(
"Cannot determine maxDirectMemory from [%s]"
,
maxDirectMemoryObj
));
}
else
{
return
((
Number
)
maxDirectMemoryObj
).
longValue
();
}
}
catch
(
ClassNotFoundException
e
)
{
throw
new
UnsupportedOperationException
(
"No VM class, cannot do memory check."
,
e
);
}
catch
(
NoSuchMethodException
e
)
{
throw
new
UnsupportedOperationException
(
"VM.maxDirectMemory doesn't exist, cannot do memory check."
,
e
);
}
catch
(
InvocationTargetException
e
)
{
throw
new
RuntimeException
(
"static method shouldn't throw this"
,
e
);
}
catch
(
IllegalAccessException
e
)
{
throw
new
RuntimeException
(
"public method, shouldn't throw this"
,
e
);
}
}
}
services/src/main/java/io/druid/cli/CliRealtimeExample.java
浏览文件 @
7547e989
...
...
@@ -42,7 +42,7 @@ import java.util.concurrent.Executor;
*/
@Command
(
name
=
"realtime"
,
description
=
"Runs a standalone realtime node for examples, see http://druid.io/docs/
0.6.73
/Realtime.html for a description"
description
=
"Runs a standalone realtime node for examples, see http://druid.io/docs/
latest
/Realtime.html for a description"
)
public
class
CliRealtimeExample
extends
ServerRunnable
{
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录