Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
2dot5
ClickHouse
提交
ce120ebc
C
ClickHouse
项目概览
2dot5
/
ClickHouse
通知
3
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
DevOps
流水线
流水线任务
计划
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
C
ClickHouse
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
DevOps
DevOps
流水线
流水线任务
计划
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
流水线任务
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
ce120ebc
编写于
8月 30, 2019
作者:
A
Alexander Kuzmenkov
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
Cosmetic change: Remove unused template parameters from Allocator.
上级
ce73e17f
变更
1
隐藏空白更改
内联
并排
Showing
1 changed file
with
38 addition
and
48 deletion
+38
-48
dbms/src/Common/Allocator.h
dbms/src/Common/Allocator.h
+38
-48
未找到文件。
dbms/src/Common/Allocator.h
浏览文件 @
ce120ebc
...
...
@@ -3,8 +3,6 @@
#include <string.h>
#ifdef NDEBUG
/// If set to 1 - randomize memory mappings manually (address space layout randomization) to reproduce more memory stomping bugs.
/// Note that Linux doesn't do it by default. This may lead to worse TLB performance.
#define ALLOCATOR_ASLR 0
#else
#define ALLOCATOR_ASLR 1
...
...
@@ -38,23 +36,27 @@
#define MAP_ANONYMOUS MAP_ANON
#endif
/** Many modern allocators (for example, tcmalloc) do not do a mremap for realloc,
*
even in case of large enough chunks of memory.
*
Although this allows
you to increase performance and reduce memory consumption during realloc.
/**
* Many modern allocators (for example, tcmalloc) do not do a mremap for
*
realloc, even in case of large enough chunks of memory. Although this allows
* you to increase performance and reduce memory consumption during realloc.
* To fix this, we do mremap manually if the chunk of memory is large enough.
* The threshold (64 MB) is chosen quite large, since changing the address space is
* very slow, especially in the case of a large number of threads.
* We expect that the set of operations mmap/something to do/mremap can only be performed about 1000 times per second.
* The threshold (64 MB) is chosen quite large, since changing the address
* space is very slow, especially in the case of a large number of threads. We
* expect that the set of operations mmap/something to do/mremap can only be
* performed about 1000 times per second.
*
* PS. This is also required, because tcmalloc can not allocate a chunk of memory greater than 16 GB.
* P.S. This is also required, because tcmalloc can not allocate a chunk of
* memory greater than 16 GB.
*/
#ifdef NDEBUG
static
constexpr
size_t
MMAP_THRESHOLD
=
64
*
(
1ULL
<<
20
);
#else
/// In debug build, use small mmap threshold to reproduce more memory stomping bugs.
/// Along with ASLR it will hopefully detect more issues than ASan.
/// The program may fail due to the limit on number of memory mappings.
/**
* In debug build, use small mmap threshold to reproduce more memory
* stomping bugs. Along with ASLR it will hopefully detect more issues than
* ASan. The program may fail due to the limit on number of memory mappings.
*/
static
constexpr
size_t
MMAP_THRESHOLD
=
4096
;
#endif
...
...
@@ -72,25 +74,6 @@ namespace ErrorCodes
}
}
namespace
AllocatorHints
{
struct
DefaultHint
{
void
*
mmap_hint
()
{
return
nullptr
;
}
};
struct
RandomHint
{
void
*
mmap_hint
()
{
return
reinterpret_cast
<
void
*>
(
std
::
uniform_int_distribution
<
intptr_t
>
(
0x100000000000UL
,
0x700000000000UL
)(
thread_local_rng
));
}
};
}
/** Responsible for allocating / freeing memory. Used, for example, in PODArray, Arena.
* Also used in hash tables.
* The interface is different from std::allocator
...
...
@@ -98,15 +81,14 @@ struct RandomHint
* - passing the size into the `free` method;
* - by the presence of the `alignment` argument;
* - the possibility of zeroing memory (used in hash tables);
* -
hint cla
ss for mmap
* -
random hint addre
ss for mmap
* - mmap_threshold for using mmap less or more
*/
template
<
bool
clear_memory_
,
typename
Hint
,
size_t
mmap_threshold
>
class
Allocator
WithHint
:
Hint
template
<
bool
clear_memory_
>
class
Allocator
{
protected:
static
constexpr
bool
clear_memory
=
clear_memory_
;
static
constexpr
size_t
small_memory_threshold
=
mmap_threshold
;
public:
/// Allocate memory range.
...
...
@@ -134,7 +116,8 @@ public:
/// nothing to do.
/// BTW, it's not possible to change alignment while doing realloc.
}
else
if
(
old_size
<
mmap_threshold
&&
new_size
<
mmap_threshold
&&
alignment
<=
MALLOC_MIN_ALIGNMENT
)
else
if
(
old_size
<
MMAP_THRESHOLD
&&
new_size
<
MMAP_THRESHOLD
&&
alignment
<=
MALLOC_MIN_ALIGNMENT
)
{
/// Resize malloc'd memory region with no special alignment requirement.
CurrentMemoryTracker
::
realloc
(
old_size
,
new_size
);
...
...
@@ -148,7 +131,7 @@ public:
if
(
new_size
>
old_size
)
memset
(
reinterpret_cast
<
char
*>
(
buf
)
+
old_size
,
0
,
new_size
-
old_size
);
}
else
if
(
old_size
>=
mmap_threshold
&&
new_size
>=
mmap_threshold
)
else
if
(
old_size
>=
MMAP_THRESHOLD
&&
new_size
>=
MMAP_THRESHOLD
)
{
/// Resize mmap'd memory region.
CurrentMemoryTracker
::
realloc
(
old_size
,
new_size
);
...
...
@@ -160,7 +143,7 @@ public:
/// No need for zero-fill, because mmap guarantees it.
}
else
if
(
new_size
<
small_memory_threshold
)
else
if
(
new_size
<
MMAP_THRESHOLD
)
{
/// Small allocs that requires a copy. Assume there's enough memory in system. Call CurrentMemoryTracker once.
CurrentMemoryTracker
::
realloc
(
old_size
,
new_size
);
...
...
@@ -194,13 +177,13 @@ private:
{
void
*
buf
;
if
(
size
>=
mmap_threshold
)
if
(
size
>=
MMAP_THRESHOLD
)
{
if
(
alignment
>
MMAP_MIN_ALIGNMENT
)
throw
DB
::
Exception
(
"Too large alignment "
+
formatReadableSizeWithBinarySuffix
(
alignment
)
+
": more than page size when allocating "
+
formatReadableSizeWithBinarySuffix
(
size
)
+
"."
,
DB
::
ErrorCodes
::
BAD_ARGUMENTS
);
buf
=
mmap
(
Hint
::
mmap_h
int
(),
size
,
PROT_READ
|
PROT_WRITE
,
MAP_PRIVATE
|
MAP_ANONYMOUS
,
-
1
,
0
);
buf
=
mmap
(
getMmapH
int
(),
size
,
PROT_READ
|
PROT_WRITE
,
MAP_PRIVATE
|
MAP_ANONYMOUS
,
-
1
,
0
);
if
(
MAP_FAILED
==
buf
)
DB
::
throwFromErrno
(
"Allocator: Cannot mmap "
+
formatReadableSizeWithBinarySuffix
(
size
)
+
"."
,
DB
::
ErrorCodes
::
CANNOT_ALLOCATE_MEMORY
);
...
...
@@ -235,7 +218,7 @@ private:
void
freeNoTrack
(
void
*
buf
,
size_t
size
)
{
if
(
size
>=
mmap_threshold
)
if
(
size
>=
MMAP_THRESHOLD
)
{
if
(
0
!=
munmap
(
buf
,
size
))
DB
::
throwFromErrno
(
"Allocator: Cannot munmap "
+
formatReadableSizeWithBinarySuffix
(
size
)
+
"."
,
DB
::
ErrorCodes
::
CANNOT_MUNMAP
);
...
...
@@ -245,15 +228,22 @@ private:
::
free
(
buf
);
}
}
};
#if ALLOCATOR_ASLR
template
<
bool
clear_memory
>
using
Allocator
=
AllocatorWithHint
<
clear_memory
,
AllocatorHints
::
RandomHint
,
MMAP_THRESHOLD
>
;
#ifndef NDEBUG
/// In debug builds, request mmap() at random addresses (a kind of ASLR), to
/// reproduce more memory stomping bugs. Note that Linux doesn't do it by
/// default. This may lead to worse TLB performance.
void
*
getMmapHint
()
{
return
reinterpret_cast
<
void
*>
(
std
::
uniform_int_distribution
<
intptr_t
>
(
0x100000000000UL
,
0x700000000000UL
)(
thread_local_rng
));
}
#else
template
<
bool
clear_memory
>
using
Allocator
=
AllocatorWithHint
<
clear_memory
,
AllocatorHints
::
DefaultHint
,
MMAP_THRESHOLD
>
;
void
*
getMmapHint
()
{
return
nullptr
;
}
#endif
};
/** When using AllocatorWithStackMemory, located on the stack,
* GCC 4.9 mistakenly assumes that we can call `free` from a pointer to the stack.
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录