Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
这是一个图像
Paddle
提交
8c9119af
P
Paddle
项目概览
这是一个图像
/
Paddle
与 Fork 源项目一致
Fork自
PaddlePaddle / Paddle
通知
1
Star
1
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
1
列表
看板
标记
里程碑
合并请求
0
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
P
Paddle
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
1
Issue
1
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
8c9119af
编写于
10月 27, 2017
作者:
G
gongweibao
提交者:
GitHub
10月 27, 2017
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
add logs and fix a bug (#5074)
add logs and fix a python path bug
上级
c1fd1dc7
变更
4
隐藏空白更改
内联
并排
Showing
4 changed file
with
25 addition
and
9 deletion
+25
-9
go/master/c/client.go
go/master/c/client.go
+2
-1
go/master/client.go
go/master/client.go
+14
-5
go/master/client_test.go
go/master/client_test.go
+1
-0
python/paddle/v2/reader/creator.py
python/paddle/v2/reader/creator.py
+8
-3
未找到文件。
go/master/c/client.go
浏览文件 @
8c9119af
...
...
@@ -123,7 +123,8 @@ func paddle_set_dataset(client C.paddle_master_client, path **C.char, size C.int
}
err
:=
c
.
SetDataset
(
paths
)
if
err
!=
nil
{
log
.
Error
(
"error set dataset"
,
log
.
Ctx
{
"error"
:
err
})
log
.
Error
(
"error set dataset"
,
log
.
Ctx
{
"error"
:
err
,
"paths"
:
paths
})
return
C
.
PADDLE_MASTER_ERROR
}
...
...
go/master/client.go
浏览文件 @
8c9119af
...
...
@@ -121,6 +121,7 @@ func (c *Client) StartGetRecords(passID int) {
}
func
(
c
*
Client
)
getRecords
(
passID
int
)
{
i
:=
0
for
{
t
,
err
:=
c
.
getTask
(
passID
)
if
err
!=
nil
{
...
...
@@ -130,12 +131,20 @@ func (c *Client) getRecords(passID int) {
c
.
ch
<-
record
{
nil
,
err
}
break
}
if
err
.
Error
()
==
ErrPassAfter
.
Error
()
{
// wait util last pass finishes
time
.
Sleep
(
time
.
Second
*
3
)
continue
if
i
%
60
==
0
{
log
.
Debug
(
"getTask of passID error."
,
log
.
Ctx
{
"error"
:
err
,
"passID"
:
passID
})
i
=
0
}
log
.
Error
(
"getTask error."
,
log
.
Ctx
{
"error"
:
err
})
// if err.Error() == ErrPassAfter.Error()
// wait util last pass finishes
// if other error such as network error
// wait to reconnect or task time out
time
.
Sleep
(
time
.
Second
*
3
)
i
+=
3
continue
}
for
_
,
chunk
:=
range
t
.
Chunks
{
...
...
go/master/client_test.go
浏览文件 @
8c9119af
...
...
@@ -117,6 +117,7 @@ func TestNextRecord(t *testing.T) {
if
e
!=
nil
{
panic
(
e
)
}
// test for n passes
for
pass
:=
0
;
pass
<
10
;
pass
++
{
c
.
StartGetRecords
(
pass
)
...
...
python/paddle/v2/reader/creator.py
浏览文件 @
8c9119af
...
...
@@ -61,7 +61,7 @@ def recordio(paths, buf_size=100):
"""
Creates a data reader from given RecordIO file paths separated by ",",
glob pattern is supported.
:path: path of recordio files.
:path: path of recordio files
, can be a string or a string list
.
:returns: data reader of recordio files.
"""
...
...
@@ -92,7 +92,7 @@ def cloud_reader(paths, etcd_endpoints, timeout_sec=5, buf_size=64):
"""
Create a data reader that yield a record one by one from
the paths:
:path
: path of recordio files
.
:path
s: path of recordio files, can be a string or a string list
.
:etcd_endpoints: the endpoints for etcd cluster
:returns: data reader of recordio files.
...
...
@@ -107,7 +107,12 @@ def cloud_reader(paths, etcd_endpoints, timeout_sec=5, buf_size=64):
import
cPickle
as
pickle
import
paddle.v2.master
as
master
c
=
master
.
client
(
etcd_endpoints
,
timeout_sec
,
buf_size
)
c
.
set_dataset
(
paths
)
if
isinstance
(
paths
,
basestring
):
path
=
[
paths
]
else
:
path
=
paths
c
.
set_dataset
(
path
)
def
reader
():
global
pass_num
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录