Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
PaddlePaddle
PALM
提交
51d7ddd8
P
PALM
项目概览
PaddlePaddle
/
PALM
通知
4
Star
3
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
10
列表
看板
标记
里程碑
合并请求
0
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
P
PALM
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
10
Issue
10
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
51d7ddd8
编写于
11月 07, 2019
作者:
X
xixiaoyao
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
fix bugs
上级
c6e33be8
变更
3
隐藏空白更改
内联
并排
Showing
3 changed file
with
14 addition
and
2 deletion
+14
-2
paddlepalm/mtl_controller.py
paddlepalm/mtl_controller.py
+4
-1
paddlepalm/reader/utils/reader4ernie.py
paddlepalm/reader/utils/reader4ernie.py
+7
-1
paddlepalm/utils/reader_helper.py
paddlepalm/utils/reader_helper.py
+3
-0
未找到文件。
paddlepalm/mtl_controller.py
浏览文件 @
51d7ddd8
...
...
@@ -13,6 +13,8 @@
# See the License for the specific language governing permissions and
# limitations under the License.
from
__future__
import
print_function
import
os
import
sys
import
importlib
...
...
@@ -397,8 +399,9 @@ class Controller(object):
# load data
for
inst
in
instances
:
print
(
inst
.
name
+
": preparing data..."
)
print
(
inst
.
name
+
": preparing data..."
,
end
=
''
)
inst
.
reader
[
'train'
].
load_data
()
print
(
'ok!'
)
# merge dataset iterators and create net input vars
iterators
=
[]
...
...
paddlepalm/reader/utils/reader4ernie.py
浏览文件 @
51d7ddd8
...
...
@@ -222,6 +222,8 @@ class BaseReader(object):
def
_prepare_batch_data
(
self
,
examples
,
batch_size
,
phase
=
None
):
"""generate batch records"""
batch_records
,
max_len
=
[],
0
if
len
(
examples
)
<
batch_size
:
raise
Exception
(
'CLS dataset contains too few samples. Expect more than '
+
str
(
batch_size
))
for
index
,
example
in
enumerate
(
examples
):
if
phase
==
"train"
:
self
.
current_example
=
index
...
...
@@ -308,7 +310,6 @@ class MaskLMReader(BaseReader):
tokens_a
=
tokenizer
.
tokenize
(
text_a
)
tokens_b
=
None
has_text_b
=
False
if
isinstance
(
example
,
dict
):
has_text_b
=
"text_b"
in
example
.
keys
()
...
...
@@ -379,6 +380,8 @@ class MaskLMReader(BaseReader):
def
batch_reader
(
self
,
examples
,
batch_size
,
in_tokens
,
phase
):
batch
=
[]
total_token_num
=
0
if
len
(
examples
)
<
batch_size
:
raise
Exception
(
'MaskLM dataset contains too few samples. Expect more than '
+
str
(
batch_size
))
for
e
in
examples
:
parsed_line
=
self
.
_convert_example_to_record
(
e
,
self
.
max_seq_len
,
self
.
tokenizer
)
to_append
=
len
(
batch
)
<
batch_size
...
...
@@ -866,6 +869,9 @@ class MRCReader(BaseReader):
"""generate batch records"""
batch_records
,
max_len
=
[],
0
if
len
(
records
)
<
batch_size
:
raise
Exception
(
'mrc dataset contains too few samples. Expect more than '
+
str
(
batch_size
))
for
index
,
record
in
enumerate
(
records
):
if
phase
==
"train"
:
self
.
current_example
=
index
...
...
paddlepalm/utils/reader_helper.py
浏览文件 @
51d7ddd8
...
...
@@ -118,10 +118,12 @@ def create_joint_iterator_fn(iterators, iterator_prefixes, joint_shape_and_dtype
results
=
_zero_batch
(
joint_shape_and_dtypes
)
outbuf
=
{}
for
id
in
task_ids
:
print
(
id
)
outputs
=
next
(
iterators
[
id
])
# dict type
outbuf
[
id
]
=
outputs
prefix
=
iterator_prefixes
[
id
]
for
outname
,
val
in
outputs
.
items
():
print
(
outname
)
task_outname
=
prefix
+
'/'
+
outname
if
outname
in
outname_to_pos
:
...
...
@@ -133,6 +135,7 @@ def create_joint_iterator_fn(iterators, iterator_prefixes, joint_shape_and_dtype
idx
=
outname_to_pos
[
task_outname
]
val
=
_check_and_adapt_shape_dtype
(
val
,
joint_shape_and_dtypes
[
idx
],
message
=
task_outname
+
': '
)
results
[
idx
]
=
val
print
(
'ok'
)
fake_batch
=
results
dev_count_bak
=
dev_count
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录