Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
曾经的那一瞬间
Models
提交
d3265962
M
Models
项目概览
曾经的那一瞬间
/
Models
大约 1 年 前同步成功
通知
1
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
DevOps
流水线
流水线任务
计划
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
M
Models
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
DevOps
DevOps
流水线
流水线任务
计划
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
流水线任务
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
d3265962
编写于
10月 26, 2020
作者:
A
Allen Wang
提交者:
A. Unique TensorFlower
10月 26, 2020
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
Add in IMDB dataset processor to TF-NLP.
PiperOrigin-RevId: 339134320
上级
4dc945c0
变更
2
隐藏空白更改
内联
并排
Showing
2 changed file
with
54 addition
and
9 deletion
+54
-9
official/nlp/data/classifier_data_lib.py
official/nlp/data/classifier_data_lib.py
+49
-6
official/nlp/data/create_finetuning_data.py
official/nlp/data/create_finetuning_data.py
+5
-3
未找到文件。
official/nlp/data/classifier_data_lib.py
浏览文件 @
d3265962
...
@@ -214,6 +214,44 @@ class ColaProcessor(DataProcessor):
...
@@ -214,6 +214,44 @@ class ColaProcessor(DataProcessor):
return
examples
return
examples
class
ImdbProcessor
(
DataProcessor
):
"""Processor for the IMDb dataset."""
def
get_labels
(
self
):
return
[
"neg"
,
"pos"
]
def
get_train_examples
(
self
,
data_dir
):
return
self
.
_create_examples
(
os
.
path
.
join
(
data_dir
,
"train"
))
def
get_dev_examples
(
self
,
data_dir
):
return
self
.
_create_examples
(
os
.
path
.
join
(
data_dir
,
"test"
))
@
staticmethod
def
get_processor_name
():
"""See base class."""
return
"IMDB"
def
_create_examples
(
self
,
data_dir
):
"""Creates examples."""
examples
=
[]
for
label
in
[
"neg"
,
"pos"
]:
cur_dir
=
os
.
path
.
join
(
data_dir
,
label
)
for
filename
in
tf
.
io
.
gfile
.
listdir
(
cur_dir
):
if
not
filename
.
endswith
(
"txt"
):
continue
if
len
(
examples
)
%
1000
==
0
:
logging
.
info
(
"Loading dev example %d"
,
len
(
examples
))
path
=
os
.
path
.
join
(
cur_dir
,
filename
)
with
tf
.
io
.
gfile
.
GFile
(
path
,
"r"
)
as
f
:
text
=
f
.
read
().
strip
().
replace
(
"<br />"
,
" "
)
examples
.
append
(
InputExample
(
guid
=
"unused_id"
,
text_a
=
text
,
text_b
=
None
,
label
=
label
))
return
examples
class
MnliProcessor
(
DataProcessor
):
class
MnliProcessor
(
DataProcessor
):
"""Processor for the MultiNLI data set (GLUE version)."""
"""Processor for the MultiNLI data set (GLUE version)."""
...
@@ -1032,6 +1070,11 @@ def convert_single_example(ex_index, example, label_list, max_seq_length,
...
@@ -1032,6 +1070,11 @@ def convert_single_example(ex_index, example, label_list, max_seq_length,
if
len
(
tokens_a
)
>
max_seq_length
-
2
:
if
len
(
tokens_a
)
>
max_seq_length
-
2
:
tokens_a
=
tokens_a
[
0
:(
max_seq_length
-
2
)]
tokens_a
=
tokens_a
[
0
:(
max_seq_length
-
2
)]
seg_id_a
=
0
seg_id_b
=
1
seg_id_cls
=
0
seg_id_pad
=
0
# The convention in BERT is:
# The convention in BERT is:
# (a) For sequence pairs:
# (a) For sequence pairs:
# tokens: [CLS] is this jack ##son ##ville ? [SEP] no it is not . [SEP]
# tokens: [CLS] is this jack ##son ##ville ? [SEP] no it is not . [SEP]
...
@@ -1053,19 +1096,19 @@ def convert_single_example(ex_index, example, label_list, max_seq_length,
...
@@ -1053,19 +1096,19 @@ def convert_single_example(ex_index, example, label_list, max_seq_length,
tokens
=
[]
tokens
=
[]
segment_ids
=
[]
segment_ids
=
[]
tokens
.
append
(
"[CLS]"
)
tokens
.
append
(
"[CLS]"
)
segment_ids
.
append
(
0
)
segment_ids
.
append
(
seg_id_cls
)
for
token
in
tokens_a
:
for
token
in
tokens_a
:
tokens
.
append
(
token
)
tokens
.
append
(
token
)
segment_ids
.
append
(
0
)
segment_ids
.
append
(
seg_id_a
)
tokens
.
append
(
"[SEP]"
)
tokens
.
append
(
"[SEP]"
)
segment_ids
.
append
(
0
)
segment_ids
.
append
(
seg_id_a
)
if
tokens_b
:
if
tokens_b
:
for
token
in
tokens_b
:
for
token
in
tokens_b
:
tokens
.
append
(
token
)
tokens
.
append
(
token
)
segment_ids
.
append
(
1
)
segment_ids
.
append
(
seg_id_b
)
tokens
.
append
(
"[SEP]"
)
tokens
.
append
(
"[SEP]"
)
segment_ids
.
append
(
1
)
segment_ids
.
append
(
seg_id_b
)
input_ids
=
tokenizer
.
convert_tokens_to_ids
(
tokens
)
input_ids
=
tokenizer
.
convert_tokens_to_ids
(
tokens
)
...
@@ -1077,7 +1120,7 @@ def convert_single_example(ex_index, example, label_list, max_seq_length,
...
@@ -1077,7 +1120,7 @@ def convert_single_example(ex_index, example, label_list, max_seq_length,
while
len
(
input_ids
)
<
max_seq_length
:
while
len
(
input_ids
)
<
max_seq_length
:
input_ids
.
append
(
0
)
input_ids
.
append
(
0
)
input_mask
.
append
(
0
)
input_mask
.
append
(
0
)
segment_ids
.
append
(
0
)
segment_ids
.
append
(
seg_id_pad
)
assert
len
(
input_ids
)
==
max_seq_length
assert
len
(
input_ids
)
==
max_seq_length
assert
len
(
input_mask
)
==
max_seq_length
assert
len
(
input_mask
)
==
max_seq_length
...
...
official/nlp/data/create_finetuning_data.py
浏览文件 @
d3265962
...
@@ -47,9 +47,9 @@ flags.DEFINE_string(
...
@@ -47,9 +47,9 @@ flags.DEFINE_string(
"for the task."
)
"for the task."
)
flags
.
DEFINE_enum
(
"classification_task_name"
,
"MNLI"
,
flags
.
DEFINE_enum
(
"classification_task_name"
,
"MNLI"
,
[
"AX"
,
"COLA"
,
"
MNLI"
,
"MRPC"
,
"PAWS-X"
,
"QNLI"
,
"QQP"
,
"RTE
"
,
[
"AX"
,
"COLA"
,
"
IMDB"
,
"MNLI"
,
"MRPC"
,
"PAWS-X"
,
"QNLI
"
,
"
SST-2"
,
"STS-B"
,
"WNLI"
,
"XNLI"
,
"XTREME-
XNLI"
,
"
QQP"
,
"RTE"
,
"SST-2"
,
"STS-B"
,
"WNLI"
,
"
XNLI"
,
"XTREME-PAWS-X"
],
"XTREME-
XNLI"
,
"XTREME-
PAWS-X"
],
"The name of the task to train BERT classifier. The "
"The name of the task to train BERT classifier. The "
"difference between XTREME-XNLI and XNLI is: 1. the format "
"difference between XTREME-XNLI and XNLI is: 1. the format "
"of input tsv files; 2. the dev set for XTREME is english "
"of input tsv files; 2. the dev set for XTREME is english "
...
@@ -182,6 +182,8 @@ def generate_classifier_dataset():
...
@@ -182,6 +182,8 @@ def generate_classifier_dataset():
classifier_data_lib
.
AxProcessor
,
classifier_data_lib
.
AxProcessor
,
"cola"
:
"cola"
:
classifier_data_lib
.
ColaProcessor
,
classifier_data_lib
.
ColaProcessor
,
"imdb"
:
classifier_data_lib
.
ImdbProcessor
,
"mnli"
:
"mnli"
:
functools
.
partial
(
classifier_data_lib
.
MnliProcessor
,
functools
.
partial
(
classifier_data_lib
.
MnliProcessor
,
mnli_type
=
FLAGS
.
mnli_type
),
mnli_type
=
FLAGS
.
mnli_type
),
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录