Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
PaddlePaddle
PaddleClas
提交
3ecefa48
P
PaddleClas
项目概览
PaddlePaddle
/
PaddleClas
1 年多 前同步成功
通知
115
Star
4999
Fork
1114
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
19
列表
看板
标记
里程碑
合并请求
6
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
P
PaddleClas
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
19
Issue
19
列表
看板
标记
里程碑
合并请求
6
合并请求
6
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
提交
3ecefa48
编写于
5月 18, 2020
作者:
W
WuHaobo
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
refine reader to support paddle1.8 API and fix the code style
上级
aab12b17
变更
1
隐藏空白更改
内联
并排
Showing
1 changed file
with
22 addition
and
23 deletion
+22
-23
ppcls/data/reader.py
ppcls/data/reader.py
+22
-23
未找到文件。
ppcls/data/reader.py
浏览文件 @
3ecefa48
#copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.
#
copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.
#
#Licensed under the Apache License, Version 2.0 (the "License");
#you may not use this file except in compliance with the License.
#You may obtain a copy of the License at
#
Licensed under the Apache License, Version 2.0 (the "License");
#
you may not use this file except in compliance with the License.
#
You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
#Unless required by applicable law or agreed to in writing, software
#distributed under the License is distributed on an "AS IS" BASIS,
#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#See the License for the specific language governing permissions and
#limitations under the License.
#
Unless required by applicable law or agreed to in writing, software
#
distributed under the License is distributed on an "AS IS" BASIS,
#
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#
See the License for the specific language governing permissions and
#
limitations under the License.
import
cv2
import
numpy
as
np
import
imghdr
import
os
import
signal
import
paddle
from
paddle.fluid.io
import
multiprocess_reader
from
.
import
imaug
from
.imaug
import
transform
from
.imaug
import
MixupOperator
from
ppcls.utils
import
logger
trainers_num
=
int
(
os
.
environ
.
get
(
'PADDLE_TRAINERS_NUM'
,
1
))
...
...
@@ -35,7 +34,7 @@ class ModeException(Exception):
def
__init__
(
self
,
message
=
''
,
mode
=
''
):
message
+=
"
\n
Only the following 3 modes are supported: "
\
"train, valid, test. Given mode is {}"
.
format
(
mode
)
"train, valid, test. Given mode is {}"
.
format
(
mode
)
super
(
ModeException
,
self
).
__init__
(
message
)
...
...
@@ -46,10 +45,10 @@ class SampleNumException(Exception):
def
__init__
(
self
,
message
=
''
,
sample_num
=
0
,
batch_size
=
1
):
message
+=
"
\n
Error: The number of the whole data ({}) "
\
"is smaller than the batch_size ({}), and drop_last "
\
"is turnning on, so nothing will feed in program, "
\
"Terminated now. Please reset batch_size to a smaller "
\
"number or feed more data!"
.
format
(
sample_num
,
batch_size
)
"is smaller than the batch_size ({}), and drop_last "
\
"is turnning on, so nothing will feed in program, "
\
"Terminated now. Please reset batch_size to a smaller "
\
"number or feed more data!"
.
format
(
sample_num
,
batch_size
)
super
(
SampleNumException
,
self
).
__init__
(
message
)
...
...
@@ -80,12 +79,12 @@ def check_params(params):
data_dir
=
params
.
get
(
'data_dir'
,
''
)
assert
os
.
path
.
isdir
(
data_dir
),
\
"{} doesn't exist, please check datadir path"
.
format
(
data_dir
)
"{} doesn't exist, please check datadir path"
.
format
(
data_dir
)
if
params
[
'mode'
]
!=
'test'
:
file_list
=
params
.
get
(
'file_list'
,
''
)
assert
os
.
path
.
isfile
(
file_list
),
\
"{} doesn't exist, please check file list path"
.
format
(
file_list
)
"{} doesn't exist, please check file list path"
.
format
(
file_list
)
def
create_file_list
(
params
):
...
...
@@ -176,8 +175,8 @@ def partial_reader(params, full_lines, part_id=0, part_num=1):
part_id(int): part index of the current partial data
part_num(int): part num of the dataset
"""
assert
part_id
<
part_num
,
(
"part_num: {} should be larger "
\
"than part_id: {}"
.
format
(
part_num
,
part_id
))
assert
part_id
<
part_num
,
(
"part_num: {} should be larger "
"than part_id: {}"
.
format
(
part_num
,
part_id
))
full_lines
=
full_lines
[
part_id
::
part_num
]
...
...
@@ -216,11 +215,11 @@ def mp_reader(params):
for
part_id
in
range
(
part_num
):
readers
.
append
(
partial_reader
(
params
,
full_lines
,
part_id
,
part_num
))
return
paddle
.
reader
.
multiprocess_reader
(
readers
,
use_pipe
=
False
)
return
multiprocess_reader
(
readers
,
use_pipe
=
False
)
def
term_mp
(
sig_num
,
frame
):
""" kill all child processes
""" kill all child processes
"""
pid
=
os
.
getpid
()
pgid
=
os
.
getpgid
(
os
.
getpid
())
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录