Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
afeixing77
Paddle
提交
1c7215ac
P
Paddle
项目概览
afeixing77
/
Paddle
与 Fork 源项目一致
Fork自
PaddlePaddle / Paddle
通知
1
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
P
Paddle
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
未验证
提交
1c7215ac
编写于
7月 08, 2020
作者:
W
WangXi
提交者:
GitHub
7月 08, 2020
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
Speedup reader check_input_array when item is array (#25395)
上级
52be62c5
变更
1
隐藏空白更改
内联
并排
Showing
1 changed file
with
13 addition
and
25 deletion
+13
-25
python/paddle/fluid/reader.py
python/paddle/fluid/reader.py
+13
-25
未找到文件。
python/paddle/fluid/reader.py
浏览文件 @
1c7215ac
...
...
@@ -97,6 +97,18 @@ class DataLoaderBase(object):
def
__next__
(
self
):
raise
NotImplementedError
()
@
classmethod
def
_check_input_array
(
cls
,
item
):
arr
=
np
.
asarray
(
item
)
if
arr
.
dtype
==
np
.
object
:
raise
TypeError
(
"
\n\t
Faild to convert input data to a regular ndarray :
\n\t
* Usually "
"this means the input data contains nested lists with different lengths. "
"
\n\t
* Check the reader function passed to 'decorate_batch_generator'"
" to locate the data causes this issue.
\n\t
* Please consider using "
"'fluid.create_lod_tensor' to convert it to a LoD-Tensor."
)
return
arr
class
DataLoader
(
object
):
"""
...
...
@@ -807,17 +819,6 @@ class DygraphGeneratorLoader(DataLoaderBase):
self
.
_reset
()
six
.
reraise
(
*
sys
.
exc_info
())
@
classmethod
def
_check_input_array
(
cls
,
item
):
arr
=
np
.
array
(
item
)
if
arr
.
dtype
==
np
.
object
:
raise
TypeError
(
"
\n\t
Faild to convert input data to a regular ndarray :
\n\t
* Usually "
"this means the input data contains nested lists with different lengths. "
"
\n\t
* Check the reader function passed to 'decorate_batch_generator'"
" to locate the data causes this issue.
\n\t
* Please consider using "
"'fluid.create_lod_tensor' to convert it to a LoD-Tensor."
)
def
_exit_thread_expectedly
(
self
):
self
.
_thread_done_event
.
set
()
self
.
_blocking_queue
.
close
()
...
...
@@ -894,7 +895,7 @@ class DygraphGeneratorLoader(DataLoaderBase):
array
=
core
.
LoDTensorArray
()
for
item
in
sample
:
if
not
isinstance
(
item
,
core
.
LoDTensor
):
self
.
_check_input_array
(
item
)
item
=
self
.
_check_input_array
(
item
)
tmp
=
core
.
LoDTensor
()
tmp
.
set
(
item
,
core
.
CPUPlace
())
item
=
tmp
...
...
@@ -1115,19 +1116,6 @@ class GeneratorLoader(DataLoaderBase):
assert
not
self
.
_iterable
,
"reset() cannot be called when DataLoader is iterable"
self
.
_reset
()
@
classmethod
def
_check_input_array
(
cls
,
item
):
arr
=
np
.
array
(
item
)
if
arr
.
dtype
==
np
.
object
:
raise
TypeError
((
"
\n\t
Faild to convert input data to a regular ndarray :
\n\t
* Usually "
"this means the input data contains nested lists with different lengths. "
"
\n\t
* Check the reader function passed to 'decorate_batch_generator'"
" to locate the data causes this issue.
\n\t
* Please consider using "
"'fluid.create_lod_tensor' to convert it to a LoD-Tensor."
))
return
arr
def
_start
(
self
):
def
__thread_main__
():
try
:
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录