Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
OPTHREE
Paddle
提交
d7b67f2b
P
Paddle
项目概览
OPTHREE
/
Paddle
与 Fork 源项目一致
Fork自
PaddlePaddle / Paddle
通知
1
Star
1
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
1
列表
看板
标记
里程碑
合并请求
0
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
P
Paddle
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
1
Issue
1
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
未验证
提交
d7b67f2b
编写于
12月 15, 2017
作者:
Y
Yancey
提交者:
GitHub
12月 15, 2017
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
fix pipe_reader on multi passes (#6627)
fix pipe reader on multi passes
上级
d40d28d8
变更
2
隐藏空白更改
内联
并排
Showing
2 changed file
with
32 addition
and
2 deletion
+32
-2
python/paddle/v2/reader/decorator.py
python/paddle/v2/reader/decorator.py
+2
-2
python/paddle/v2/reader/tests/decorator_test.py
python/paddle/v2/reader/tests/decorator_test.py
+30
-0
未找到文件。
python/paddle/v2/reader/decorator.py
浏览文件 @
d7b67f2b
...
@@ -390,8 +390,6 @@ def pipe_reader(left_cmd,
...
@@ -390,8 +390,6 @@ def pipe_reader(left_cmd,
if
not
callable
(
parser
):
if
not
callable
(
parser
):
raise
TypeError
(
"parser must be a callable object"
)
raise
TypeError
(
"parser must be a callable object"
)
process
=
subprocess
.
Popen
(
left_cmd
.
split
(
" "
),
bufsize
=
bufsize
,
stdout
=
subprocess
.
PIPE
)
# TODO(typhoonzero): add a thread to read stderr
# TODO(typhoonzero): add a thread to read stderr
# Always init a decompress object is better than
# Always init a decompress object is better than
...
@@ -400,6 +398,8 @@ def pipe_reader(left_cmd,
...
@@ -400,6 +398,8 @@ def pipe_reader(left_cmd,
32
+
zlib
.
MAX_WBITS
)
# offset 32 to skip the header
32
+
zlib
.
MAX_WBITS
)
# offset 32 to skip the header
def
reader
():
def
reader
():
process
=
subprocess
.
Popen
(
left_cmd
.
split
(
" "
),
bufsize
=
bufsize
,
stdout
=
subprocess
.
PIPE
)
remained
=
""
remained
=
""
while
True
:
while
True
:
buff
=
process
.
stdout
.
read
(
bufsize
)
buff
=
process
.
stdout
.
read
(
bufsize
)
...
...
python/paddle/v2/reader/tests/decorator_test.py
浏览文件 @
d7b67f2b
...
@@ -145,5 +145,35 @@ class TestXmap(unittest.TestCase):
...
@@ -145,5 +145,35 @@ class TestXmap(unittest.TestCase):
self
.
assertEqual
(
e
,
mapper
(
idx
))
self
.
assertEqual
(
e
,
mapper
(
idx
))
class
TestPipeReader
(
unittest
.
TestCase
):
def
test_pipe_reader
(
self
):
def
simple_parser
(
lines
):
return
lines
import
tempfile
records
=
[
str
(
i
)
for
i
in
xrange
(
5
)]
temp
=
tempfile
.
NamedTemporaryFile
()
try
:
with
open
(
temp
.
name
,
'w'
)
as
f
:
for
r
in
records
:
f
.
write
(
'%s
\n
'
%
r
)
cmd
=
"cat %s"
%
temp
.
name
reader
=
paddle
.
v2
.
reader
.
pipe_reader
(
cmd
,
simple_parser
,
bufsize
=
128
)
for
i
in
xrange
(
4
):
result
=
[]
for
r
in
reader
():
result
.
append
(
r
)
for
idx
,
e
in
enumerate
(
records
):
print
e
,
result
[
idx
]
self
.
assertEqual
(
e
,
result
[
idx
])
finally
:
# delete the temporary file
temp
.
close
()
if
__name__
==
'__main__'
:
if
__name__
==
'__main__'
:
unittest
.
main
()
unittest
.
main
()
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录