Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
机器未来
Paddle
提交
ea548a79
P
Paddle
项目概览
机器未来
/
Paddle
与 Fork 源项目一致
Fork自
PaddlePaddle / Paddle
通知
1
Star
1
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
1
列表
看板
标记
里程碑
合并请求
0
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
P
Paddle
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
1
Issue
1
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
提交
ea548a79
编写于
7月 26, 2018
作者:
C
chenweihang
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
refactor: simplify class to function
上级
999d097b
变更
1
隐藏空白更改
内联
并排
Showing
1 changed file
with
54 addition
and
51 deletion
+54
-51
python/paddle/fluid/contrib/memory_usage_calc.py
python/paddle/fluid/contrib/memory_usage_calc.py
+54
-51
未找到文件。
python/paddle/fluid/contrib/memory_usage_calc.py
浏览文件 @
ea548a79
...
...
@@ -23,9 +23,7 @@ This API is still under active development and may change drastically.
from
..
import
core
from
..framework
import
Program
,
Variable
__all__
=
[
'MemoryInfo'
]
DEBUG
=
False
__all__
=
[
'memory_usage'
]
dtype_to_size
=
{
core
.
VarDesc
.
VarType
.
FP16
:
2
,
...
...
@@ -38,62 +36,67 @@ dtype_to_size = {
core
.
VarDesc
.
VarType
.
UINT8
:
1
,
}
DEBUG
=
False
class
MemoryInfo
(
object
):
def
__init__
(
self
,
program
):
if
not
isinstance
(
program
,
Program
):
raise
TypeError
(
"Calculating Memory Usage requires Program as its Parameter."
"But you passed in %s"
%
(
type
(
prgram
)))
self
.
_program
=
program
def
_has_var
(
self
,
block
,
var_name
):
return
block
.
has_var
(
str
(
var_name
))
def
memory_usage
(
program
,
batch_size
):
"""
Get the estimate memory usage of program with input batch size.
def
_find_var
(
self
,
block
,
var_name
):
return
block
.
var
(
str
(
var_name
))
Args:
program(Program): The current Program.
batch_size(int): The current input data batch_size.
Returns:
min_total_memory(float): the estimate memory usage lower bound.
max_total_memory(float): the estimate memory usage upper bound.
unit_str(string): the unit of estimate usage result.
Examples:
>>> import paddle.fluid as fluid
>>> lower_usage, upper_usage, unit = fluid.contrib.memory_usage(
fluid.default_main_program(), batch_size=10)
>>> print "memory usage is about %.3f - %.3f %s" %
\
(lower_usage, upper_usage, unit)
def
get_memory_usage
(
self
,
batch_size
,
with_details
=
False
):
"""
# get the first block of program
first_block
=
self
.
_program
.
global_block
()
# Parameters check
if
not
isinstance
(
program
,
Program
):
raise
TypeError
(
"Calculating Memory Usage requires Program as its Parameter."
"But you passed in %s"
%
(
type
(
prgram
)))
if
batch_size
<=
0
:
raise
ValueError
(
"The batch size need to be positive."
)
# get the var_name list of first block
# TODO(chenweihang): not find the API get block's var list directly
total_memory
=
0.0
for
var
in
self
.
_program
.
list_vars
():
if
DEBUG
:
print
"All Block's Var: %s"
%
(
var
.
name
)
# TODO(chenweihang): why not used program.list_vars()
# calculate all variable's memory directly?
if
self
.
_has_var
(
first_block
,
var
.
name
):
if
DEBUG
:
print
"First Block's Var: %s"
%
(
var
.
name
)
print
"Var's shape: "
,
var
.
shape
print
"Var's dtype: "
,
var
.
dtype
data_count
=
1
for
x
in
var
.
shape
:
if
x
==
-
1
:
data_count
*=
batch_size
else
:
data_count
*=
x
var_memory
=
data_count
*
dtype_to_size
[
var
.
dtype
]
if
DEBUG
:
print
"Var's memory: %d"
%
(
var_memory
)
total_memory
+=
var_memory
# Get the var_name list of first block and calculate
total_memory
=
0.0
for
var
in
program
.
global_block
().
vars
.
itervalues
():
data_count
=
1
for
x
in
var
.
shape
:
if
x
==
-
1
:
data_count
*=
batch_size
else
:
data_count
*=
x
var_memory
=
data_count
*
dtype_to_size
[
var
.
dtype
]
if
DEBUG
:
print
"%s memory usage: %d"
%
(
var
.
name
,
var_memory
)
total_memory
+=
var_memory
if
DEBUG
:
print
"total memory usage: %.2f"
%
(
total_memory
)
# Convert unit and make result string
result_str
=
"- With current batch size, memory usage is about "
unit_str
=
" B."
# Convert appropriate unit
unit_str
=
"B"
if
total_memory
>
1024
:
total_memory
/=
1024
unit_str
=
"KB"
if
total_memory
>
1024
:
total_memory
/=
1024
unit_str
=
" KB."
if
total_memory
>
1024
:
total_memory
/=
1024
unit_str
=
" MB."
unit_str
=
"MB"
# Append extra memory consumption (5% - 10%)
result_str
+=
str
(
round
(
total_memory
*
1.05
,
3
))
+
" - "
\
+
str
(
round
(
total_memory
*
1.10
,
3
))
+
unit_str
# Append extra memory consumption (5% - 10%)
min_total_memory
=
total_memory
*
1.05
max_total_memory
=
total_memory
*
1.1
return
resul
t_str
return
min_total_memory
,
max_total_memory
,
uni
t_str
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录