' \
+ --predict_file gen_data/wmt16_ende_data_bpe/newstest2014.tok.bpe.32000.en-de \
+ --batch_size 32 \
+ --init_from_params trained_params/step_100000 \
+ --beam_size 5 \
+ --max_out_len 255 \
+ --output_file predict.txt \
+ --n_head 16 \
+ --d_model 1024 \
+ --d_inner_hid 4096 \
+ --prepostprocess_dropout 0.3
+```
+
+
+### 模型评估
+
+预测结果中每行输出是对应行输入的得分最高的翻译，对于使用 BPE 的数据，预测出的翻译结果也将是 BPE 表示的数据，要还原成原始的数据（这里指 tokenize 后的数据）才能进行正确的评估。评估过程具体如下（BLEU 是翻译任务常用的自动评估方法指标）：
+
+```sh
+# 还原 predict.txt 中的预测结果为 tokenize 后的数据
+sed -r 's/(@@ )|(@@ ?$)//g' predict.txt > predict.tok.txt
+# 若无 BLEU 评估工具，需先进行下载
+# git clone https://github.com/moses-smt/mosesdecoder.git
+# 以英德翻译 newstest2014 测试数据为例
+perl gen_data/mosesdecoder/scripts/generic/multi-bleu.perl gen_data/wmt16_ende_data/newstest2014.tok.de < predict.tok.txt
+```
+可以看到类似如下的结果：
+```
+BLEU = 26.35, 57.7/32.1/20.0/13.0 (BP=1.000, ratio=1.013, hyp_len=63903, ref_len=63078)
+```
+
+使用本项目中提供的内容，英德翻译 base model 和 big model 八卡训练 100K 个 iteration 后测试有大约如下的 BLEU 值：
+
+| 测试集 | newstest2014 | newstest2015 | newstest2016 |
+|-|-|-|-|
+| Base | 26.35 | 29.07 | 33.30 |
+| Big | 27.07 | 30.09 | 34.38 |
+
+### 预训练模型
+
+我们这里提供了对应有以上 BLEU 值的 [base model](https://transformer-res.bj.bcebos.com/base_model_dygraph.tar.gz) 和 [big model](https://transformer-res.bj.bcebos.com/big_model_dygraph.tar.gz) 的模型参数提供下载使用（注意，模型使用了提供下载的数据进行训练和测试）。
+
+## 进阶使用
+
+### 背景介绍
+
+Transformer 是论文 [Attention Is All You Need](https://arxiv.org/abs/1706.03762) 中提出的用以完成机器翻译（machine translation, MT）等序列到序列（sequence to sequence, Seq2Seq）学习任务的一种全新网络结构，其完全使用注意力（Attention）机制来实现序列到序列的建模[1]。
+
+相较于此前 Seq2Seq 模型中广泛使用的循环神经网络（Recurrent Neural Network, RNN），使用（Self）Attention 进行输入序列到输出序列的变换主要具有以下优势：
+
+- 计算复杂度小
+ - 特征维度为 d 、长度为 n 的序列，在 RNN 中计算复杂度为 `O(n * d * d)` （n 个时间步，每个时间步计算 d 维的矩阵向量乘法），在 Self-Attention 中计算复杂度为 `O(n * n * d)` （n 个时间步两两计算 d 维的向量点积或其他相关度函数），n 通常要小于 d 。
+- 计算并行度高
+ - RNN 中当前时间步的计算要依赖前一个时间步的计算结果；Self-Attention 中各时间步的计算只依赖输入不依赖之前时间步输出，各时间步可以完全并行。
+- 容易学习长程依赖（long-range dependencies）
+ - RNN 中相距为 n 的两个位置间的关联需要 n 步才能建立；Self-Attention 中任何两个位置都直接相连；路径越短信号传播越容易。
+
+Transformer 中引入使用的基于 Self-Attention 的序列建模模块结构，已被广泛应用在 Bert [2]等语义表示模型中，取得了显著效果。
+
+
+### 模型概览
+
+Transformer 同样使用了 Seq2Seq 模型中典型的编码器-解码器（Encoder-Decoder）的框架结构，整体网络结构如图1所示。
+
+
+
+图 1. Transformer 网络结构图
+
+
+可以看到，和以往 Seq2Seq 模型不同，Transformer 的 Encoder 和 Decoder 中不再使用 RNN 的结构。
+
+### 模型特点
+
+Transformer 中的 Encoder 由若干相同的 layer 堆叠组成，每个 layer 主要由多头注意力（Multi-Head Attention）和全连接的前馈（Feed-Forward）网络这两个 sub-layer 构成。
+- Multi-Head Attention 在这里用于实现 Self-Attention，相比于简单的 Attention 机制，其将输入进行多路线性变换后分别计算 Attention 的结果，并将所有结果拼接后再次进行线性变换作为输出。参见图2，其中 Attention 使用的是点积（Dot-Product），并在点积后进行了 scale 的处理以避免因点积结果过大进入 softmax 的饱和区域。
+- Feed-Forward 网络会对序列中的每个位置进行相同的计算（Position-wise），其采用的是两次线性变换中间加以 ReLU 激活的结构。
+
+此外，每个 sub-layer 后还施以 Residual Connection [3]和 Layer Normalization [4]来促进梯度传播和模型收敛。
+
+
+
+图 2. Multi-Head Attention
+
+
+Decoder 具有和 Encoder 类似的结构，只是相比于组成 Encoder 的 layer ，在组成 Decoder 的 layer 中还多了一个 Multi-Head Attention 的 sub-layer 来实现对 Encoder 输出的 Attention，这个 Encoder-Decoder Attention 在其他 Seq2Seq 模型中也是存在的。
+
+## FAQ
+
+**Q:** 预测结果中样本数少于输入的样本数是什么原因
+**A:** 若样本中最大长度超过 `transformer.yaml` 中 `max_length` 的默认设置，请注意运行时增大 `--max_length` 的设置，否则超长样本将被过滤。
+
+**Q:** 预测时最大长度超过了训练时的最大长度怎么办
+**A:** 由于训练时 `max_length` 的设置决定了保存模型 position encoding 的大小，若预测时长度超过 `max_length`，请调大该值，会重新生成更大的 position encoding 表。
+
+
+## 参考文献
+1. Vaswani A, Shazeer N, Parmar N, et al. [Attention is all you need](http://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf)[C]//Advances in Neural Information Processing Systems. 2017: 6000-6010.
+2. Devlin J, Chang M W, Lee K, et al. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://arxiv.org/abs/1810.04805)[J]. arXiv preprint arXiv:1810.04805, 2018.
+3. He K, Zhang X, Ren S, et al. [Deep residual learning for image recognition](http://openaccess.thecvf.com/content_cvpr_2016/papers/He_Deep_Residual_Learning_CVPR_2016_paper.pdf)[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 770-778.
+4. Ba J L, Kiros J R, Hinton G E. [Layer normalization](https://arxiv.org/pdf/1607.06450.pdf)[J]. arXiv preprint arXiv:1607.06450, 2016.
+5. Sennrich R, Haddow B, Birch A. [Neural machine translation of rare words with subword units](https://arxiv.org/pdf/1508.07909)[J]. arXiv preprint arXiv:1508.07909, 2015.
+
+
+## 作者
+- [guochengCS](https://github.com/guoshengCS)
+
+## 如何贡献代码
+
+如果你可以修复某个issue或者增加一个新功能，欢迎给我们提交PR。如果对应的PR被接受了，我们将根据贡献的质量和难度进行打分（0-5分，越高越好）。如果你累计获得了10分，可以联系我们获得面试机会或者为你写推荐信。
diff --git a/transformer/images/multi_head_attention.png b/transformer/images/multi_head_attention.png
new file mode 100644
index 0000000000000000000000000000000000000000..427fb6b32aaeb7013066a167aab4fb97c024c2d6
Binary files /dev/null and b/transformer/images/multi_head_attention.png differ
diff --git a/transformer/images/transformer_network.png b/transformer/images/transformer_network.png
new file mode 100644
index 0000000000000000000000000000000000000000..34be0e5c7e2b08f858683d86353db5e81049c7ca
Binary files /dev/null and b/transformer/images/transformer_network.png differ
diff --git a/transformer/predict.py b/transformer/predict.py
new file mode 100644
index 0000000000000000000000000000000000000000..ec9c5629f9e5ebc7198f009d7d05df263d640281
--- /dev/null
+++ b/transformer/predict.py
@@ -0,0 +1,150 @@
+# Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import logging
+import os
+import six
+import sys
+sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+from functools import partial
+
+import numpy as np
+import paddle
+import paddle.fluid as fluid
+from paddle.fluid.io import DataLoader
+from paddle.fluid.layers.utils import flatten
+
+from utils.configure import PDConfig
+from utils.check import check_gpu, check_version
+
+from model import Input, set_device
+from reader import prepare_infer_input, Seq2SeqDataset, Seq2SeqBatchSampler
+from transformer import InferTransformer, position_encoding_init
+
+
+def post_process_seq(seq, bos_idx, eos_idx, output_bos=False,
+ output_eos=False):
+ """
+ Post-process the decoded sequence.
+ """
+ eos_pos = len(seq) - 1
+ for i, idx in enumerate(seq):
+ if idx == eos_idx:
+ eos_pos = i
+ break
+ seq = [
+ idx for idx in seq[:eos_pos + 1]
+ if (output_bos or idx != bos_idx) and (output_eos or idx != eos_idx)
+ ]
+ return seq
+
+
+def do_predict(args):
+ device = set_device("gpu" if args.use_cuda else "cpu")
+ fluid.enable_dygraph(device) if args.eager_run else None
+
+ inputs = [
+ Input(
+ [None, None], "int64", name="src_word"),
+ Input(
+ [None, None], "int64", name="src_pos"),
+ Input(
+ [None, args.n_head, None, None],
+ "float32",
+ name="src_slf_attn_bias"),
+ Input(
+ [None, args.n_head, None, None],
+ "float32",
+ name="trg_src_attn_bias"),
+ ]
+
+ # define data
+ dataset = Seq2SeqDataset(
+ fpattern=args.predict_file,
+ src_vocab_fpath=args.src_vocab_fpath,
+ trg_vocab_fpath=args.trg_vocab_fpath,
+ token_delimiter=args.token_delimiter,
+ start_mark=args.special_token[0],
+ end_mark=args.special_token[1],
+ unk_mark=args.special_token[2])
+ args.src_vocab_size, args.trg_vocab_size, args.bos_idx, args.eos_idx, \
+ args.unk_idx = dataset.get_vocab_summary()
+ trg_idx2word = Seq2SeqDataset.load_dict(
+ dict_path=args.trg_vocab_fpath, reverse=True)
+ batch_sampler = Seq2SeqBatchSampler(
+ dataset=dataset,
+ use_token_batch=False,
+ batch_size=args.batch_size,
+ max_length=args.max_length)
+ data_loader = DataLoader(
+ dataset=dataset,
+ batch_sampler=batch_sampler,
+ places=device,
+ feed_list=None
+ if fluid.in_dygraph_mode() else [x.forward() for x in inputs],
+ collate_fn=partial(
+ prepare_infer_input, src_pad_idx=args.eos_idx, n_head=args.n_head),
+ num_workers=0,
+ return_list=True)
+
+ # define model
+ transformer = InferTransformer(
+ args.src_vocab_size,
+ args.trg_vocab_size,
+ args.max_length + 1,
+ args.n_layer,
+ args.n_head,
+ args.d_key,
+ args.d_value,
+ args.d_model,
+ args.d_inner_hid,
+ args.prepostprocess_dropout,
+ args.attention_dropout,
+ args.relu_dropout,
+ args.preprocess_cmd,
+ args.postprocess_cmd,
+ args.weight_sharing,
+ args.bos_idx,
+ args.eos_idx,
+ beam_size=args.beam_size,
+ max_out_len=args.max_out_len)
+ transformer.prepare(inputs=inputs)
+
+ # load the trained model
+ assert args.init_from_params, (
+ "Please set init_from_params to load the infer model.")
+ transformer.load(os.path.join(args.init_from_params, "transformer"))
+
+ # TODO: use model.predict when support variant length
+ f = open(args.output_file, "wb")
+ for data in data_loader():
+ finished_seq = transformer.test(inputs=flatten(data))[0]
+ finished_seq = np.transpose(finished_seq, [0, 2, 1])
+ for ins in finished_seq:
+ for beam_idx, beam in enumerate(ins):
+ if beam_idx >= args.n_best: break
+ id_list = post_process_seq(beam, args.bos_idx, args.eos_idx)
+ word_list = [trg_idx2word[id] for id in id_list]
+ sequence = b" ".join(word_list) + b"\n"
+ f.write(sequence)
+
+
+if __name__ == "__main__":
+ args = PDConfig(yaml_file="./transformer.yaml")
+ args.build()
+ args.Print()
+ check_gpu(args.use_cuda)
+ check_version()
+
+ do_predict(args)
diff --git a/transformer/reader.py b/transformer/reader.py
new file mode 100644
index 0000000000000000000000000000000000000000..b83617d7179b65d9e2bc335eec1f226423a08fdb
--- /dev/null
+++ b/transformer/reader.py
@@ -0,0 +1,423 @@
+# Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import glob
+import six
+import os
+import tarfile
+import itertools
+
+import numpy as np
+import paddle.fluid as fluid
+from paddle.fluid.dygraph.parallel import ParallelEnv
+from paddle.fluid.io import BatchSampler, DataLoader, Dataset
+
+
+def prepare_train_input(insts, src_pad_idx, trg_pad_idx, n_head):
+ """
+ Put all padded data needed by training into a list.
+ """
+ src_word, src_pos, src_slf_attn_bias, src_max_len = pad_batch_data(
+ [inst[0] for inst in insts], src_pad_idx, n_head, is_target=False)
+ src_word = src_word.reshape(-1, src_max_len)
+ src_pos = src_pos.reshape(-1, src_max_len)
+ trg_word, trg_pos, trg_slf_attn_bias, trg_max_len = pad_batch_data(
+ [inst[1] for inst in insts], trg_pad_idx, n_head, is_target=True)
+ trg_word = trg_word.reshape(-1, trg_max_len)
+ trg_pos = trg_pos.reshape(-1, trg_max_len)
+
+ trg_src_attn_bias = np.tile(src_slf_attn_bias[:, :, ::src_max_len, :],
+ [1, 1, trg_max_len, 1]).astype("float32")
+
+ lbl_word, lbl_weight, num_token = pad_batch_data(
+ [inst[2] for inst in insts],
+ trg_pad_idx,
+ n_head,
+ is_target=False,
+ is_label=True,
+ return_attn_bias=False,
+ return_max_len=False,
+ return_num_token=True)
+ lbl_word = lbl_word.reshape(-1, 1)
+ lbl_weight = lbl_weight.reshape(-1, 1)
+
+ data_inputs = [
+ src_word, src_pos, src_slf_attn_bias, trg_word, trg_pos,
+ trg_slf_attn_bias, trg_src_attn_bias, lbl_word, lbl_weight
+ ]
+
+ return data_inputs
+
+
+def prepare_infer_input(insts, src_pad_idx, n_head):
+ """
+ Put all padded data needed by beam search decoder into a list.
+ """
+ src_word, src_pos, src_slf_attn_bias, src_max_len = pad_batch_data(
+ [inst[0] for inst in insts], src_pad_idx, n_head, is_target=False)
+ trg_src_attn_bias = np.tile(src_slf_attn_bias[:, :, ::src_max_len, :],
+ [1, 1, 1, 1]).astype("float32")
+ src_word = src_word.reshape(-1, src_max_len)
+ src_pos = src_pos.reshape(-1, src_max_len)
+
+ data_inputs = [
+ src_word, src_pos, src_slf_attn_bias, trg_src_attn_bias
+ ]
+ return data_inputs
+
+
+def pad_batch_data(insts,
+ pad_idx,
+ n_head,
+ is_target=False,
+ is_label=False,
+ return_attn_bias=True,
+ return_max_len=True,
+ return_num_token=False):
+ """
+ Pad the instances to the max sequence length in batch, and generate the
+ corresponding position data and attention bias.
+ """
+ return_list = []
+ max_len = max(len(inst) for inst in insts)
+ # Any token included in dict can be used to pad, since the paddings' loss
+ # will be masked out by weights and make no effect on parameter gradients.
+ inst_data = np.array(
+ [inst + [pad_idx] * (max_len - len(inst)) for inst in insts])
+ return_list += [inst_data.astype("int64").reshape([-1, 1])]
+ if is_label: # label weight
+ inst_weight = np.array([[1.] * len(inst) + [0.] * (max_len - len(inst))
+ for inst in insts])
+ return_list += [inst_weight.astype("float32").reshape([-1, 1])]
+ else: # position data
+ inst_pos = np.array([
+ list(range(0, len(inst))) + [0] * (max_len - len(inst))
+ for inst in insts
+ ])
+ return_list += [inst_pos.astype("int64").reshape([-1, 1])]
+ if return_attn_bias:
+ if is_target:
+ # This is used to avoid attention on paddings and subsequent
+ # words.
+ slf_attn_bias_data = np.ones(
+ (inst_data.shape[0], max_len, max_len))
+ slf_attn_bias_data = np.triu(slf_attn_bias_data,
+ 1).reshape([-1, 1, max_len, max_len])
+ slf_attn_bias_data = np.tile(slf_attn_bias_data,
+ [1, n_head, 1, 1]) * [-1e9]
+ else:
+ # This is used to avoid attention on paddings.
+ slf_attn_bias_data = np.array([[0] * len(inst) + [-1e9] *
+ (max_len - len(inst))
+ for inst in insts])
+ slf_attn_bias_data = np.tile(
+ slf_attn_bias_data.reshape([-1, 1, 1, max_len]),
+ [1, n_head, max_len, 1])
+ return_list += [slf_attn_bias_data.astype("float32")]
+ if return_max_len:
+ return_list += [max_len]
+ if return_num_token:
+ num_token = 0
+ for inst in insts:
+ num_token += len(inst)
+ return_list += [num_token]
+ return return_list if len(return_list) > 1 else return_list[0]
+
+
+class SortType(object):
+ GLOBAL = 'global'
+ POOL = 'pool'
+ NONE = "none"
+
+
+class Converter(object):
+ def __init__(self, vocab, beg, end, unk, delimiter, add_beg):
+ self._vocab = vocab
+ self._beg = beg
+ self._end = end
+ self._unk = unk
+ self._delimiter = delimiter
+ self._add_beg = add_beg
+
+ def __call__(self, sentence):
+ return ([self._beg] if self._add_beg else []) + [
+ self._vocab.get(w, self._unk)
+ for w in sentence.split(self._delimiter)
+ ] + [self._end]
+
+
+class ComposedConverter(object):
+ def __init__(self, converters):
+ self._converters = converters
+
+ def __call__(self, parallel_sentence):
+ return [
+ self._converters[i](parallel_sentence[i])
+ for i in range(len(self._converters))
+ ]
+
+
+class SentenceBatchCreator(object):
+ def __init__(self, batch_size):
+ self.batch = []
+ self._batch_size = batch_size
+
+ def append(self, info):
+ self.batch.append(info)
+ if len(self.batch) == self._batch_size:
+ tmp = self.batch
+ self.batch = []
+ return tmp
+
+
+class TokenBatchCreator(object):
+ def __init__(self, batch_size):
+ self.batch = []
+ self.max_len = -1
+ self._batch_size = batch_size
+
+ def append(self, info):
+ cur_len = info.max_len
+ max_len = max(self.max_len, cur_len)
+ if max_len * (len(self.batch) + 1) > self._batch_size:
+ result = self.batch
+ self.batch = [info]
+ self.max_len = cur_len
+ return result
+ else:
+ self.max_len = max_len
+ self.batch.append(info)
+
+
+class SampleInfo(object):
+ def __init__(self, i, max_len, min_len):
+ self.i = i
+ self.min_len = min_len
+ self.max_len = max_len
+
+
+class MinMaxFilter(object):
+ def __init__(self, max_len, min_len, underlying_creator):
+ self._min_len = min_len
+ self._max_len = max_len
+ self._creator = underlying_creator
+
+ def append(self, info):
+ if info.max_len > self._max_len or info.min_len < self._min_len:
+ return
+ else:
+ return self._creator.append(info)
+
+ @property
+ def batch(self):
+ return self._creator.batch
+
+
+class Seq2SeqDataset(Dataset):
+ def __init__(self,
+ src_vocab_fpath,
+ trg_vocab_fpath,
+ fpattern,
+ tar_fname=None,
+ field_delimiter="\t",
+ token_delimiter=" ",
+ start_mark="",
+ end_mark="",
+ unk_mark="",
+ only_src=False):
+ # convert str to bytes, and use byte data
+ field_delimiter = field_delimiter.encode("utf8")
+ token_delimiter = token_delimiter.encode("utf8")
+ start_mark = start_mark.encode("utf8")
+ end_mark = end_mark.encode("utf8")
+ unk_mark = unk_mark.encode("utf8")
+ self._src_vocab = self.load_dict(src_vocab_fpath)
+ self._trg_vocab = self.load_dict(trg_vocab_fpath)
+ self._bos_idx = self._src_vocab[start_mark]
+ self._eos_idx = self._src_vocab[end_mark]
+ self._unk_idx = self._src_vocab[unk_mark]
+ self._only_src = only_src
+ self._field_delimiter = field_delimiter
+ self._token_delimiter = token_delimiter
+ self.load_src_trg_ids(fpattern, tar_fname)
+
+ def load_src_trg_ids(self, fpattern, tar_fname):
+ converters = [
+ Converter(vocab=self._src_vocab,
+ beg=self._bos_idx,
+ end=self._eos_idx,
+ unk=self._unk_idx,
+ delimiter=self._token_delimiter,
+ add_beg=False)
+ ]
+ if not self._only_src:
+ converters.append(
+ Converter(vocab=self._trg_vocab,
+ beg=self._bos_idx,
+ end=self._eos_idx,
+ unk=self._unk_idx,
+ delimiter=self._token_delimiter,
+ add_beg=True))
+
+ converters = ComposedConverter(converters)
+
+ self._src_seq_ids = []
+ self._trg_seq_ids = None if self._only_src else []
+ self._sample_infos = []
+
+ for i, line in enumerate(self._load_lines(fpattern, tar_fname)):
+ src_trg_ids = converters(line)
+ self._src_seq_ids.append(src_trg_ids[0])
+ lens = [len(src_trg_ids[0])]
+ if not self._only_src:
+ self._trg_seq_ids.append(src_trg_ids[1])
+ lens.append(len(src_trg_ids[1]))
+ self._sample_infos.append(SampleInfo(i, max(lens), min(lens)))
+
+ def _load_lines(self, fpattern, tar_fname):
+ fpaths = glob.glob(fpattern)
+ assert len(fpaths) > 0, "no matching file to the provided data path"
+
+ if len(fpaths) == 1 and tarfile.is_tarfile(fpaths[0]):
+ if tar_fname is None:
+ raise Exception("If tar file provided, please set tar_fname.")
+
+ f = tarfile.open(fpaths[0], "rb")
+ for line in f.extractfile(tar_fname):
+ fields = line.strip(b"\n").split(self._field_delimiter)
+ if (not self._only_src
+ and len(fields) == 2) or (self._only_src
+ and len(fields) == 1):
+ yield fields
+ else:
+ for fpath in fpaths:
+ if not os.path.isfile(fpath):
+ raise IOError("Invalid file: %s" % fpath)
+
+ with open(fpath, "rb") as f:
+ for line in f:
+ fields = line.strip(b"\n").split(self._field_delimiter)
+ if (not self._only_src and len(fields) == 2) or (
+ self._only_src and len(fields) == 1):
+ yield fields
+
+ @staticmethod
+ def load_dict(dict_path, reverse=False):
+ word_dict = {}
+ with open(dict_path, "rb") as fdict:
+ for idx, line in enumerate(fdict):
+ if reverse:
+ word_dict[idx] = line.strip(b"\n")
+ else:
+ word_dict[line.strip(b"\n")] = idx
+ return word_dict
+
+ def get_vocab_summary(self):
+ return len(self._src_vocab), len(
+ self._trg_vocab), self._bos_idx, self._eos_idx, self._unk_idx
+
+ def __getitem__(self, idx):
+ return (self._src_seq_ids[idx], self._trg_seq_ids[idx][:-1],
+ self._trg_seq_ids[idx][1:]
+ ) if not self._only_src else self._src_seq_ids[idx]
+
+ def __len__(self):
+ return len(self._sample_infos)
+
+
+class Seq2SeqBatchSampler(BatchSampler):
+ def __init__(self,
+ dataset,
+ batch_size,
+ pool_size=10000,
+ sort_type=SortType.NONE,
+ min_length=0,
+ max_length=100,
+ shuffle=False,
+ shuffle_batch=False,
+ use_token_batch=False,
+ clip_last_batch=False,
+ seed=0):
+ for arg, value in locals().items():
+ if arg != "self":
+ setattr(self, "_" + arg, value)
+ self._random = np.random
+ self._random.seed(seed)
+ # for multi-devices
+ self._nranks = ParallelEnv().nranks
+ self._local_rank = ParallelEnv().local_rank
+ self._device_id = ParallelEnv().dev_id
+
+ def __iter__(self):
+ # global sort or global shuffle
+ if self._sort_type == SortType.GLOBAL:
+ infos = sorted(self._dataset._sample_infos,
+ key=lambda x: x.max_len)
+ else:
+ if self._shuffle:
+ infos = self._dataset._sample_infos
+ self._random.shuffle(infos)
+ else:
+ infos = self._dataset._sample_infos
+
+ if self._sort_type == SortType.POOL:
+ reverse = True
+ for i in range(0, len(infos), self._pool_size):
+ # to avoid placing short next to long sentences
+ reverse = not reverse
+ infos[i:i + self._pool_size] = sorted(
+ infos[i:i + self._pool_size],
+ key=lambda x: x.max_len,
+ reverse=reverse)
+
+ batches = []
+ batch_creator = TokenBatchCreator(
+ self._batch_size
+ ) if self._use_token_batch else SentenceBatchCreator(self._batch_size *
+ self._nranks)
+ batch_creator = MinMaxFilter(self._max_length, self._min_length,
+ batch_creator)
+
+ for info in infos:
+ batch = batch_creator.append(info)
+ if batch is not None:
+ batches.append(batch)
+
+ if not self._clip_last_batch and len(batch_creator.batch) != 0:
+ batches.append(batch_creator.batch)
+
+ if self._shuffle_batch:
+ self._random.shuffle(batches)
+
+ if not self._use_token_batch:
+ # when producing batches according to sequence number, to confirm
+ # neighbor batches which would be feed and run parallel have similar
+ # length (thus similar computational cost) after shuffle, we as take
+ # them as a whole when shuffling and split here
+ batches = [[
+ batch[self._batch_size * i:self._batch_size * (i + 1)]
+ for i in range(self._nranks)
+ ] for batch in batches]
+ batches = list(itertools.chain.from_iterable(batches))
+
+ # for multi-device
+ for batch_id, batch in enumerate(batches):
+ if batch_id % self._nranks == self._local_rank:
+ batch_indices = [info.i for info in batch]
+ yield batch_indices
+ if self._local_rank > len(batches) % self._nranks:
+ yield batch_indices
+
+ def __len__(self):
+ return 100
diff --git a/transformer/run.sh b/transformer/run.sh
new file mode 100644
index 0000000000000000000000000000000000000000..a55d7a7ac747636c2c4fcdfbec1a0f5160a7be05
--- /dev/null
+++ b/transformer/run.sh
@@ -0,0 +1,41 @@
+python -u train.py \
+ --epoch 30 \
+ --src_vocab_fpath wmt16_ende_data_bpe/vocab_all.bpe.32000 \
+ --trg_vocab_fpath wmt16_ende_data_bpe/vocab_all.bpe.32000 \
+ --special_token '' '' '' \
+ --training_file wmt16_ende_data_bpe/train.tok.clean.bpe.32000.en-de.tiny \
+ --validation_file wmt16_ende_data_bpe/newstest2014.tok.bpe.32000.en-de \
+ --batch_size 4096 \
+ --print_step 1 \
+ --use_cuda True \
+ --random_seed 1000 \
+ --save_step 10 \
+ --eager_run True
+ #--init_from_pretrain_model base_model_dygraph/step_100000/ \
+ #--init_from_checkpoint trained_models/step_200/transformer
+ #--n_head 16 \
+ #--d_model 1024 \
+ #--d_inner_hid 4096 \
+ #--prepostprocess_dropout 0.3
+exit
+
+echo `date`
+
+python -u predict.py \
+ --src_vocab_fpath wmt16_ende_data_bpe/vocab_all.bpe.32000 \
+ --trg_vocab_fpath wmt16_ende_data_bpe/vocab_all.bpe.32000 \
+ --special_token '' '' '' \
+ --predict_file wmt16_ende_data_bpe/newstest2014.tok.bpe.32000.en-de \
+ --batch_size 64 \
+ --init_from_params base_model_dygraph/step_100000/ \
+ --beam_size 5 \
+ --max_out_len 255 \
+ --output_file predict.txt \
+ --eager_run True
+ #--max_length 500 \
+ #--n_head 16 \
+ #--d_model 1024 \
+ #--d_inner_hid 4096 \
+ #--prepostprocess_dropout 0.3
+
+echo `date`
\ No newline at end of file
diff --git a/transformer/train.py b/transformer/train.py
new file mode 100644
index 0000000000000000000000000000000000000000..557241843b0d039ddc3344d0f6eae6a855af83b9
--- /dev/null
+++ b/transformer/train.py
@@ -0,0 +1,199 @@
+# Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import logging
+import os
+import six
+import sys
+sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+from functools import partial
+
+import numpy as np
+import paddle
+import paddle.fluid as fluid
+from paddle.fluid.dygraph import to_variable
+from paddle.fluid.io import DataLoader
+
+from utils.configure import PDConfig
+from utils.check import check_gpu, check_version
+
+from model import Input, set_device
+from callbacks import ProgBarLogger
+from reader import prepare_train_input, Seq2SeqDataset, Seq2SeqBatchSampler
+from transformer import Transformer, CrossEntropyCriterion, NoamDecay
+
+
+class LoggerCallback(ProgBarLogger):
+ def __init__(self, log_freq=1, verbose=2, loss_normalizer=0.):
+ super(LoggerCallback, self).__init__(log_freq, verbose)
+ # TODO: wrap these override function to simplify
+ self.loss_normalizer = loss_normalizer
+
+ def on_train_begin(self, logs=None):
+ super(LoggerCallback, self).on_train_begin(logs)
+ self.train_metrics += ["normalized loss", "ppl"]
+
+ def on_train_batch_end(self, step, logs=None):
+ logs["normalized loss"] = logs["loss"][0] - self.loss_normalizer
+ logs["ppl"] = np.exp(min(logs["loss"][0], 100))
+ super(LoggerCallback, self).on_train_batch_end(step, logs)
+
+ def on_eval_begin(self, logs=None):
+ super(LoggerCallback, self).on_eval_begin(logs)
+ self.eval_metrics += ["normalized loss", "ppl"]
+
+ def on_eval_batch_end(self, step, logs=None):
+ logs["normalized loss"] = logs["loss"][0] - self.loss_normalizer
+ logs["ppl"] = np.exp(min(logs["loss"][0], 100))
+ super(LoggerCallback, self).on_eval_batch_end(step, logs)
+
+
+def do_train(args):
+ device = set_device("gpu" if args.use_cuda else "cpu")
+ fluid.enable_dygraph(device) if args.eager_run else None
+
+ # set seed for CE
+ random_seed = eval(str(args.random_seed))
+ if random_seed is not None:
+ fluid.default_main_program().random_seed = random_seed
+ fluid.default_startup_program().random_seed = random_seed
+
+ # define inputs
+ inputs = [
+ Input(
+ [None, None], "int64", name="src_word"),
+ Input(
+ [None, None], "int64", name="src_pos"),
+ Input(
+ [None, args.n_head, None, None],
+ "float32",
+ name="src_slf_attn_bias"),
+ Input(
+ [None, None], "int64", name="trg_word"),
+ Input(
+ [None, None], "int64", name="trg_pos"),
+ Input(
+ [None, args.n_head, None, None],
+ "float32",
+ name="trg_slf_attn_bias"),
+ Input(
+ [None, args.n_head, None, None],
+ "float32",
+ name="trg_src_attn_bias"),
+ ]
+ labels = [
+ Input(
+ [None, 1], "int64", name="label"),
+ Input(
+ [None, 1], "float32", name="weight"),
+ ]
+
+ # def dataloader
+ data_loaders = [None, None]
+ data_files = [args.training_file, args.validation_file
+ ] if args.validation_file else [args.training_file]
+ for i, data_file in enumerate(data_files):
+ dataset = Seq2SeqDataset(
+ fpattern=data_file,
+ src_vocab_fpath=args.src_vocab_fpath,
+ trg_vocab_fpath=args.trg_vocab_fpath,
+ token_delimiter=args.token_delimiter,
+ start_mark=args.special_token[0],
+ end_mark=args.special_token[1],
+ unk_mark=args.special_token[2])
+ args.src_vocab_size, args.trg_vocab_size, args.bos_idx, args.eos_idx, \
+ args.unk_idx = dataset.get_vocab_summary()
+ batch_sampler = Seq2SeqBatchSampler(
+ dataset=dataset,
+ use_token_batch=args.use_token_batch,
+ batch_size=args.batch_size,
+ pool_size=args.pool_size,
+ sort_type=args.sort_type,
+ shuffle=args.shuffle,
+ shuffle_batch=args.shuffle_batch,
+ max_length=args.max_length)
+ data_loader = DataLoader(
+ dataset=dataset,
+ batch_sampler=batch_sampler,
+ places=device,
+ feed_list=None if fluid.in_dygraph_mode() else
+ [x.forward() for x in inputs + labels],
+ collate_fn=partial(
+ prepare_train_input,
+ src_pad_idx=args.eos_idx,
+ trg_pad_idx=args.eos_idx,
+ n_head=args.n_head),
+ num_workers=0, # TODO: use multi-process
+ return_list=True)
+ data_loaders[i] = data_loader
+ train_loader, eval_loader = data_loaders
+
+ # define model
+ transformer = Transformer(
+ args.src_vocab_size, args.trg_vocab_size, args.max_length + 1,
+ args.n_layer, args.n_head, args.d_key, args.d_value, args.d_model,
+ args.d_inner_hid, args.prepostprocess_dropout, args.attention_dropout,
+ args.relu_dropout, args.preprocess_cmd, args.postprocess_cmd,
+ args.weight_sharing, args.bos_idx, args.eos_idx)
+
+ transformer.prepare(
+ fluid.optimizer.Adam(
+ learning_rate=fluid.layers.noam_decay(args.d_model,
+ args.warmup_steps),
+ beta1=args.beta1,
+ beta2=args.beta2,
+ epsilon=float(args.eps),
+ parameter_list=transformer.parameters()),
+ CrossEntropyCriterion(args.label_smooth_eps),
+ inputs=inputs,
+ labels=labels)
+
+ ## init from some checkpoint, to resume the previous training
+ if args.init_from_checkpoint:
+ transformer.load(
+ os.path.join(args.init_from_checkpoint, "transformer"))
+ ## init from some pretrain models, to better solve the current task
+ if args.init_from_pretrain_model:
+ transformer.load(
+ os.path.join(args.init_from_pretrain_model, "transformer"),
+ reset_optimizer=True)
+
+ # the best cross-entropy value with label smoothing
+ loss_normalizer = -(
+ (1. - args.label_smooth_eps) * np.log(
+ (1. - args.label_smooth_eps)) + args.label_smooth_eps *
+ np.log(args.label_smooth_eps / (args.trg_vocab_size - 1) + 1e-20))
+
+ # model train
+ transformer.fit(train_data=train_loader,
+ eval_data=eval_loader,
+ epochs=1,
+ eval_freq=1,
+ save_freq=1,
+ verbose=2,
+ callbacks=[
+ LoggerCallback(
+ log_freq=args.print_step,
+ loss_normalizer=loss_normalizer)
+ ])
+
+
+if __name__ == "__main__":
+ args = PDConfig(yaml_file="./transformer.yaml")
+ args.build()
+ args.Print()
+ check_gpu(args.use_cuda)
+ check_version()
+
+ do_train(args)
diff --git a/transformer/transformer.py b/transformer/transformer.py
new file mode 100644
index 0000000000000000000000000000000000000000..fd64759c1c03b9f6cf6f38990b7a9f4522b3d409
--- /dev/null
+++ b/transformer/transformer.py
@@ -0,0 +1,691 @@
+# Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import print_function
+
+import numpy as np
+
+import paddle.fluid as fluid
+import paddle.fluid.layers as layers
+from paddle.fluid.dygraph import Embedding, LayerNorm, Linear, Layer, to_variable
+from paddle.fluid.dygraph.learning_rate_scheduler import LearningRateDecay
+from model import Model, CrossEntropy, Loss
+from text import TransformerBeamSearchDecoder, DynamicDecode
+
+
+def position_encoding_init(n_position, d_pos_vec):
+ """
+ Generate the initial values for the sinusoid position encoding table.
+ """
+ channels = d_pos_vec
+ position = np.arange(n_position)
+ num_timescales = channels // 2
+ log_timescale_increment = (np.log(float(1e4) / float(1)) /
+ (num_timescales - 1))
+ inv_timescales = np.exp(np.arange(
+ num_timescales)) * -log_timescale_increment
+ scaled_time = np.expand_dims(position, 1) * np.expand_dims(inv_timescales,
+ 0)
+ signal = np.concatenate([np.sin(scaled_time), np.cos(scaled_time)], axis=1)
+ signal = np.pad(signal, [[0, 0], [0, np.mod(channels, 2)]], 'constant')
+ position_enc = signal
+ return position_enc.astype("float32")
+
+
+class NoamDecay(LearningRateDecay):
+ """
+ learning rate scheduler
+ """
+
+ def __init__(self,
+ d_model,
+ warmup_steps,
+ static_lr=2.0,
+ begin=1,
+ step=1,
+ dtype='float32'):
+ super(NoamDecay, self).__init__(begin, step, dtype)
+ self.d_model = d_model
+ self.warmup_steps = warmup_steps
+ self.static_lr = static_lr
+
+ def step(self):
+ a = self.create_lr_var(self.step_num**-0.5)
+ b = self.create_lr_var((self.warmup_steps**-1.5) * self.step_num)
+ lr_value = (self.d_model**-0.5) * layers.elementwise_min(
+ a, b) * self.static_lr
+ return lr_value
+
+
+class PrePostProcessLayer(Layer):
+ """
+ PrePostProcessLayer
+ """
+
+ def __init__(self, process_cmd, d_model, dropout_rate):
+ super(PrePostProcessLayer, self).__init__()
+ self.process_cmd = process_cmd
+ self.functors = []
+ for cmd in self.process_cmd:
+ if cmd == "a": # add residual connection
+ self.functors.append(lambda x, y: x + y if y else x)
+ elif cmd == "n": # add layer normalization
+ self.functors.append(
+ self.add_sublayer(
+ "layer_norm_%d" % len(
+ self.sublayers(include_sublayers=False)),
+ LayerNorm(
+ normalized_shape=d_model,
+ param_attr=fluid.ParamAttr(
+ initializer=fluid.initializer.Constant(1.)),
+ bias_attr=fluid.ParamAttr(
+ initializer=fluid.initializer.Constant(0.)))))
+ elif cmd == "d": # add dropout
+ self.functors.append(lambda x: layers.dropout(
+ x, dropout_prob=dropout_rate, is_test=False)
+ if dropout_rate else x)
+
+ def forward(self, x, residual=None):
+ for i, cmd in enumerate(self.process_cmd):
+ if cmd == "a":
+ x = self.functors[i](x, residual)
+ else:
+ x = self.functors[i](x)
+ return x
+
+
+class MultiHeadAttention(Layer):
+ """
+ Multi-Head Attention
+ """
+
+ def __init__(self, d_key, d_value, d_model, n_head=1, dropout_rate=0.):
+ super(MultiHeadAttention, self).__init__()
+ self.n_head = n_head
+ self.d_key = d_key
+ self.d_value = d_value
+ self.d_model = d_model
+ self.dropout_rate = dropout_rate
+ self.q_fc = Linear(
+ input_dim=d_model, output_dim=d_key * n_head, bias_attr=False)
+ self.k_fc = Linear(
+ input_dim=d_model, output_dim=d_key * n_head, bias_attr=False)
+ self.v_fc = Linear(
+ input_dim=d_model, output_dim=d_value * n_head, bias_attr=False)
+ self.proj_fc = Linear(
+ input_dim=d_value * n_head, output_dim=d_model, bias_attr=False)
+
+ def _prepare_qkv(self, queries, keys, values, cache=None):
+ if keys is None: # self-attention
+ keys, values = queries, queries
+ static_kv = False
+ else: # cross-attention
+ static_kv = True
+
+ q = self.q_fc(queries)
+ q = layers.reshape(x=q, shape=[0, 0, self.n_head, self.d_key])
+ q = layers.transpose(x=q, perm=[0, 2, 1, 3])
+
+ if cache is not None and static_kv and "static_k" in cache:
+ # for encoder-decoder attention in inference and has cached
+ k = cache["static_k"]
+ v = cache["static_v"]
+ else:
+ k = self.k_fc(keys)
+ v = self.v_fc(values)
+ k = layers.reshape(x=k, shape=[0, 0, self.n_head, self.d_key])
+ k = layers.transpose(x=k, perm=[0, 2, 1, 3])
+ v = layers.reshape(x=v, shape=[0, 0, self.n_head, self.d_value])
+ v = layers.transpose(x=v, perm=[0, 2, 1, 3])
+
+ if cache is not None:
+ if static_kv and not "static_k" in cache:
+ # for encoder-decoder attention in inference and has not cached
+ cache["static_k"], cache["static_v"] = k, v
+ elif not static_kv:
+ # for decoder self-attention in inference
+ cache_k, cache_v = cache["k"], cache["v"]
+ k = layers.concat([cache_k, k], axis=2)
+ v = layers.concat([cache_v, v], axis=2)
+ cache["k"], cache["v"] = k, v
+
+ return q, k, v
+
+ def forward(self, queries, keys, values, attn_bias, cache=None):
+ # compute q ,k ,v
+ q, k, v = self._prepare_qkv(queries, keys, values, cache)
+
+ # scale dot product attention
+ product = layers.matmul(
+ x=q, y=k, transpose_y=True, alpha=self.d_model**-0.5)
+ if attn_bias:
+ product += attn_bias
+ weights = layers.softmax(product)
+ if self.dropout_rate:
+ weights = layers.dropout(
+ weights, dropout_prob=self.dropout_rate, is_test=False)
+
+ out = layers.matmul(weights, v)
+
+ # combine heads
+ out = layers.transpose(out, perm=[0, 2, 1, 3])
+ out = layers.reshape(x=out, shape=[0, 0, out.shape[2] * out.shape[3]])
+
+ # project to output
+ out = self.proj_fc(out)
+ return out
+
+ def cal_kv(self, keys, values):
+ k = self.k_fc(keys)
+ v = self.v_fc(values)
+ k = layers.reshape(x=k, shape=[0, 0, self.n_head, self.d_key])
+ k = layers.transpose(x=k, perm=[0, 2, 1, 3])
+ v = layers.reshape(x=v, shape=[0, 0, self.n_head, self.d_value])
+ v = layers.transpose(x=v, perm=[0, 2, 1, 3])
+ return k, v
+
+
+class FFN(Layer):
+ """
+ Feed-Forward Network
+ """
+
+ def __init__(self, d_inner_hid, d_model, dropout_rate):
+ super(FFN, self).__init__()
+ self.dropout_rate = dropout_rate
+ self.fc1 = Linear(
+ input_dim=d_model, output_dim=d_inner_hid, act="relu")
+ self.fc2 = Linear(input_dim=d_inner_hid, output_dim=d_model)
+
+ def forward(self, x):
+ hidden = self.fc1(x)
+ if self.dropout_rate:
+ hidden = layers.dropout(
+ hidden, dropout_prob=self.dropout_rate, is_test=False)
+ out = self.fc2(hidden)
+ return out
+
+
+class EncoderLayer(Layer):
+ """
+ EncoderLayer
+ """
+
+ def __init__(self,
+ n_head,
+ d_key,
+ d_value,
+ d_model,
+ d_inner_hid,
+ prepostprocess_dropout,
+ attention_dropout,
+ relu_dropout,
+ preprocess_cmd="n",
+ postprocess_cmd="da"):
+
+ super(EncoderLayer, self).__init__()
+
+ self.preprocesser1 = PrePostProcessLayer(preprocess_cmd, d_model,
+ prepostprocess_dropout)
+ self.self_attn = MultiHeadAttention(d_key, d_value, d_model, n_head,
+ attention_dropout)
+ self.postprocesser1 = PrePostProcessLayer(postprocess_cmd, d_model,
+ prepostprocess_dropout)
+
+ self.preprocesser2 = PrePostProcessLayer(preprocess_cmd, d_model,
+ prepostprocess_dropout)
+ self.ffn = FFN(d_inner_hid, d_model, relu_dropout)
+ self.postprocesser2 = PrePostProcessLayer(postprocess_cmd, d_model,
+ prepostprocess_dropout)
+
+ def forward(self, enc_input, attn_bias):
+ attn_output = self.self_attn(
+ self.preprocesser1(enc_input), None, None, attn_bias)
+ attn_output = self.postprocesser1(attn_output, enc_input)
+
+ ffn_output = self.ffn(self.preprocesser2(attn_output))
+ ffn_output = self.postprocesser2(ffn_output, attn_output)
+ return ffn_output
+
+
+class Encoder(Layer):
+ """
+ encoder
+ """
+
+ def __init__(self,
+ n_layer,
+ n_head,
+ d_key,
+ d_value,
+ d_model,
+ d_inner_hid,
+ prepostprocess_dropout,
+ attention_dropout,
+ relu_dropout,
+ preprocess_cmd="n",
+ postprocess_cmd="da"):
+
+ super(Encoder, self).__init__()
+
+ self.encoder_layers = list()
+ for i in range(n_layer):
+ self.encoder_layers.append(
+ self.add_sublayer(
+ "layer_%d" % i,
+ EncoderLayer(n_head, d_key, d_value, d_model, d_inner_hid,
+ prepostprocess_dropout, attention_dropout,
+ relu_dropout, preprocess_cmd,
+ postprocess_cmd)))
+ self.processer = PrePostProcessLayer(preprocess_cmd, d_model,
+ prepostprocess_dropout)
+
+ def forward(self, enc_input, attn_bias):
+ for encoder_layer in self.encoder_layers:
+ enc_output = encoder_layer(enc_input, attn_bias)
+ enc_input = enc_output
+
+ return self.processer(enc_output)
+
+
+class Embedder(Layer):
+ """
+ Word Embedding + Position Encoding
+ """
+
+ def __init__(self, vocab_size, emb_dim, bos_idx=0):
+ super(Embedder, self).__init__()
+
+ self.word_embedder = Embedding(
+ size=[vocab_size, emb_dim],
+ padding_idx=bos_idx,
+ param_attr=fluid.ParamAttr(
+ initializer=fluid.initializer.Normal(0., emb_dim**-0.5)))
+
+ def forward(self, word):
+ word_emb = self.word_embedder(word)
+ return word_emb
+
+
+class WrapEncoder(Layer):
+ """
+ embedder + encoder
+ """
+
+ def __init__(self, src_vocab_size, max_length, n_layer, n_head, d_key,
+ d_value, d_model, d_inner_hid, prepostprocess_dropout,
+ attention_dropout, relu_dropout, preprocess_cmd,
+ postprocess_cmd, word_embedder):
+ super(WrapEncoder, self).__init__()
+
+ self.emb_dropout = prepostprocess_dropout
+ self.emb_dim = d_model
+ self.word_embedder = word_embedder
+ self.pos_encoder = Embedding(
+ size=[max_length, self.emb_dim],
+ param_attr=fluid.ParamAttr(
+ initializer=fluid.initializer.NumpyArrayInitializer(
+ position_encoding_init(max_length, self.emb_dim)),
+ trainable=False))
+
+ self.encoder = Encoder(n_layer, n_head, d_key, d_value, d_model,
+ d_inner_hid, prepostprocess_dropout,
+ attention_dropout, relu_dropout, preprocess_cmd,
+ postprocess_cmd)
+
+ def forward(self, src_word, src_pos, src_slf_attn_bias):
+ word_emb = self.word_embedder(src_word)
+ word_emb = layers.scale(x=word_emb, scale=self.emb_dim**0.5)
+ pos_enc = self.pos_encoder(src_pos)
+ pos_enc.stop_gradient = True
+ emb = word_emb + pos_enc
+ enc_input = layers.dropout(
+ emb, dropout_prob=self.emb_dropout,
+ is_test=False) if self.emb_dropout else emb
+
+ enc_output = self.encoder(enc_input, src_slf_attn_bias)
+ return enc_output
+
+
+class DecoderLayer(Layer):
+ """
+ decoder
+ """
+
+ def __init__(self,
+ n_head,
+ d_key,
+ d_value,
+ d_model,
+ d_inner_hid,
+ prepostprocess_dropout,
+ attention_dropout,
+ relu_dropout,
+ preprocess_cmd="n",
+ postprocess_cmd="da"):
+ super(DecoderLayer, self).__init__()
+
+ self.preprocesser1 = PrePostProcessLayer(preprocess_cmd, d_model,
+ prepostprocess_dropout)
+ self.self_attn = MultiHeadAttention(d_key, d_value, d_model, n_head,
+ attention_dropout)
+ self.postprocesser1 = PrePostProcessLayer(postprocess_cmd, d_model,
+ prepostprocess_dropout)
+
+ self.preprocesser2 = PrePostProcessLayer(preprocess_cmd, d_model,
+ prepostprocess_dropout)
+ self.cross_attn = MultiHeadAttention(d_key, d_value, d_model, n_head,
+ attention_dropout)
+ self.postprocesser2 = PrePostProcessLayer(postprocess_cmd, d_model,
+ prepostprocess_dropout)
+
+ self.preprocesser3 = PrePostProcessLayer(preprocess_cmd, d_model,
+ prepostprocess_dropout)
+ self.ffn = FFN(d_inner_hid, d_model, relu_dropout)
+ self.postprocesser3 = PrePostProcessLayer(postprocess_cmd, d_model,
+ prepostprocess_dropout)
+
+ def forward(self,
+ dec_input,
+ enc_output,
+ self_attn_bias,
+ cross_attn_bias,
+ cache=None):
+ self_attn_output = self.self_attn(
+ self.preprocesser1(dec_input), None, None, self_attn_bias, cache)
+ self_attn_output = self.postprocesser1(self_attn_output, dec_input)
+
+ cross_attn_output = self.cross_attn(
+ self.preprocesser2(self_attn_output), enc_output, enc_output,
+ cross_attn_bias, cache)
+ cross_attn_output = self.postprocesser2(cross_attn_output,
+ self_attn_output)
+
+ ffn_output = self.ffn(self.preprocesser3(cross_attn_output))
+ ffn_output = self.postprocesser3(ffn_output, cross_attn_output)
+
+ return ffn_output
+
+
+class Decoder(Layer):
+ """
+ decoder
+ """
+
+ def __init__(self, n_layer, n_head, d_key, d_value, d_model, d_inner_hid,
+ prepostprocess_dropout, attention_dropout, relu_dropout,
+ preprocess_cmd, postprocess_cmd):
+ super(Decoder, self).__init__()
+
+ self.decoder_layers = list()
+ for i in range(n_layer):
+ self.decoder_layers.append(
+ self.add_sublayer(
+ "layer_%d" % i,
+ DecoderLayer(n_head, d_key, d_value, d_model, d_inner_hid,
+ prepostprocess_dropout, attention_dropout,
+ relu_dropout, preprocess_cmd,
+ postprocess_cmd)))
+ self.processer = PrePostProcessLayer(preprocess_cmd, d_model,
+ prepostprocess_dropout)
+
+ def forward(self,
+ dec_input,
+ enc_output,
+ self_attn_bias,
+ cross_attn_bias,
+ caches=None):
+ for i, decoder_layer in enumerate(self.decoder_layers):
+ dec_output = decoder_layer(dec_input, enc_output, self_attn_bias,
+ cross_attn_bias, None
+ if caches is None else caches[i])
+ dec_input = dec_output
+
+ return self.processer(dec_output)
+
+ def prepare_static_cache(self, enc_output):
+ return [
+ dict(
+ zip(("static_k", "static_v"),
+ decoder_layer.cross_attn.cal_kv(enc_output, enc_output)))
+ for decoder_layer in self.decoder_layers
+ ]
+
+
+class WrapDecoder(Layer):
+ """
+ embedder + decoder
+ """
+
+ def __init__(self, trg_vocab_size, max_length, n_layer, n_head, d_key,
+ d_value, d_model, d_inner_hid, prepostprocess_dropout,
+ attention_dropout, relu_dropout, preprocess_cmd,
+ postprocess_cmd, share_input_output_embed, word_embedder):
+ super(WrapDecoder, self).__init__()
+
+ self.emb_dropout = prepostprocess_dropout
+ self.emb_dim = d_model
+ self.word_embedder = word_embedder
+ self.pos_encoder = Embedding(
+ size=[max_length, self.emb_dim],
+ param_attr=fluid.ParamAttr(
+ initializer=fluid.initializer.NumpyArrayInitializer(
+ position_encoding_init(max_length, self.emb_dim)),
+ trainable=False))
+
+ self.decoder = Decoder(n_layer, n_head, d_key, d_value, d_model,
+ d_inner_hid, prepostprocess_dropout,
+ attention_dropout, relu_dropout, preprocess_cmd,
+ postprocess_cmd)
+
+ if share_input_output_embed:
+ self.linear = lambda x: layers.matmul(x=x,
+ y=self.word_embedder.
+ word_embedder.weight,
+ transpose_y=True)
+ else:
+ self.linear = Linear(
+ input_dim=d_model, output_dim=trg_vocab_size, bias_attr=False)
+
+ def forward(self,
+ trg_word,
+ trg_pos,
+ trg_slf_attn_bias,
+ trg_src_attn_bias,
+ enc_output,
+ caches=None):
+ word_emb = self.word_embedder(trg_word)
+ word_emb = layers.scale(x=word_emb, scale=self.emb_dim**0.5)
+ pos_enc = self.pos_encoder(trg_pos)
+ pos_enc.stop_gradient = True
+ emb = word_emb + pos_enc
+ dec_input = layers.dropout(
+ emb, dropout_prob=self.emb_dropout,
+ is_test=False) if self.emb_dropout else emb
+ dec_output = self.decoder(dec_input, enc_output, trg_slf_attn_bias,
+ trg_src_attn_bias, caches)
+ dec_output = layers.reshape(
+ dec_output,
+ shape=[-1, dec_output.shape[-1]], )
+ logits = self.linear(dec_output)
+ return logits
+
+
+class CrossEntropyCriterion(Loss):
+ def __init__(self, label_smooth_eps):
+ super(CrossEntropyCriterion, self).__init__()
+ self.label_smooth_eps = label_smooth_eps
+
+ def forward(self, outputs, labels):
+ predict, (label, weights) = outputs[0], labels
+ if self.label_smooth_eps:
+ label = layers.label_smooth(
+ label=layers.one_hot(
+ input=label, depth=predict.shape[-1]),
+ epsilon=self.label_smooth_eps)
+
+ cost = layers.softmax_with_cross_entropy(
+ logits=predict,
+ label=label,
+ soft_label=True if self.label_smooth_eps else False)
+ weighted_cost = cost * weights
+ sum_cost = layers.reduce_sum(weighted_cost)
+ token_num = layers.reduce_sum(weights)
+ token_num.stop_gradient = True
+ avg_cost = sum_cost / token_num
+ return avg_cost
+
+
+class Transformer(Model):
+ """
+ model
+ """
+
+ def __init__(self,
+ src_vocab_size,
+ trg_vocab_size,
+ max_length,
+ n_layer,
+ n_head,
+ d_key,
+ d_value,
+ d_model,
+ d_inner_hid,
+ prepostprocess_dropout,
+ attention_dropout,
+ relu_dropout,
+ preprocess_cmd,
+ postprocess_cmd,
+ weight_sharing,
+ bos_id=0,
+ eos_id=1):
+ super(Transformer, self).__init__()
+ src_word_embedder = Embedder(
+ vocab_size=src_vocab_size, emb_dim=d_model, bos_idx=bos_id)
+ self.encoder = WrapEncoder(
+ src_vocab_size, max_length, n_layer, n_head, d_key, d_value,
+ d_model, d_inner_hid, prepostprocess_dropout, attention_dropout,
+ relu_dropout, preprocess_cmd, postprocess_cmd, src_word_embedder)
+ if weight_sharing:
+ assert src_vocab_size == trg_vocab_size, (
+ "Vocabularies in source and target should be same for weight sharing."
+ )
+ trg_word_embedder = src_word_embedder
+ else:
+ trg_word_embedder = Embedder(
+ vocab_size=trg_vocab_size, emb_dim=d_model, bos_idx=bos_id)
+ self.decoder = WrapDecoder(
+ trg_vocab_size, max_length, n_layer, n_head, d_key, d_value,
+ d_model, d_inner_hid, prepostprocess_dropout, attention_dropout,
+ relu_dropout, preprocess_cmd, postprocess_cmd, weight_sharing,
+ trg_word_embedder)
+
+ self.trg_vocab_size = trg_vocab_size
+ self.n_layer = n_layer
+ self.n_head = n_head
+ self.d_key = d_key
+ self.d_value = d_value
+
+ def forward(self, src_word, src_pos, src_slf_attn_bias, trg_word, trg_pos,
+ trg_slf_attn_bias, trg_src_attn_bias):
+ enc_output = self.encoder(src_word, src_pos, src_slf_attn_bias)
+ predict = self.decoder(trg_word, trg_pos, trg_slf_attn_bias,
+ trg_src_attn_bias, enc_output)
+ return predict
+
+
+class TransfomerCell(object):
+ """
+ Let inputs=(trg_word, trg_pos), states=cache to make Transformer can be
+ used as RNNCell
+ """
+
+ def __init__(self, decoder):
+ self.decoder = decoder
+
+ def __call__(self, inputs, states, trg_src_attn_bias, enc_output,
+ static_caches):
+ trg_word, trg_pos = inputs
+ for cache, static_cache in zip(states, static_caches):
+ cache.update(static_cache)
+ logits = self.decoder(trg_word, trg_pos, None, trg_src_attn_bias,
+ enc_output, states)
+ new_states = [{"k": cache["k"], "v": cache["v"]} for cache in states]
+ return logits, new_states
+
+
+class InferTransformer(Transformer):
+ """
+ model for prediction
+ """
+
+ def __init__(self,
+ src_vocab_size,
+ trg_vocab_size,
+ max_length,
+ n_layer,
+ n_head,
+ d_key,
+ d_value,
+ d_model,
+ d_inner_hid,
+ prepostprocess_dropout,
+ attention_dropout,
+ relu_dropout,
+ preprocess_cmd,
+ postprocess_cmd,
+ weight_sharing,
+ bos_id=0,
+ eos_id=1,
+ beam_size=4,
+ max_out_len=256):
+ args = dict(locals())
+ args.pop("self")
+ args.pop("__class__", None) # py3
+ self.beam_size = args.pop("beam_size")
+ self.max_out_len = args.pop("max_out_len")
+ super(InferTransformer, self).__init__(**args)
+ cell = TransfomerCell(self.decoder)
+ self.beam_search_decoder = DynamicDecode(
+ TransformerBeamSearchDecoder(
+ cell, bos_id, eos_id, beam_size, var_dim_in_state=2),
+ max_out_len,
+ is_test=True)
+
+ def forward(self, src_word, src_pos, src_slf_attn_bias, trg_src_attn_bias):
+ enc_output = self.encoder(src_word, src_pos, src_slf_attn_bias)
+ ## init states (caches) for transformer, need to be updated according to selected beam
+ caches = [{
+ "k": layers.fill_constant_batch_size_like(
+ input=enc_output,
+ shape=[-1, self.n_head, 0, self.d_key],
+ dtype=enc_output.dtype,
+ value=0),
+ "v": layers.fill_constant_batch_size_like(
+ input=enc_output,
+ shape=[-1, self.n_head, 0, self.d_value],
+ dtype=enc_output.dtype,
+ value=0),
+ } for i in range(self.n_layer)]
+ enc_output = TransformerBeamSearchDecoder.tile_beam_merge_with_batch(
+ enc_output, self.beam_size)
+ trg_src_attn_bias = TransformerBeamSearchDecoder.tile_beam_merge_with_batch(
+ trg_src_attn_bias, self.beam_size)
+ static_caches = self.decoder.decoder.prepare_static_cache(enc_output)
+ rs, _ = self.beam_search_decoder(
+ inits=caches,
+ enc_output=enc_output,
+ trg_src_attn_bias=trg_src_attn_bias,
+ static_caches=static_caches)
+ return rs
diff --git a/transformer/transformer.yaml b/transformer/transformer.yaml
new file mode 100644
index 0000000000000000000000000000000000000000..58bc72ba3d4ed6e94969778e34e4e1cea812ed57
--- /dev/null
+++ b/transformer/transformer.yaml
@@ -0,0 +1,112 @@
+# used for continuous evaluation
+enable_ce: False
+
+eager_run: False
+
+# The frequency to save trained models when training.
+save_step: 10000
+# The frequency to fetch and print output when training.
+print_step: 100
+# path of the checkpoint, to resume the previous training
+init_from_checkpoint: ""
+# path of the pretrain model, to better solve the current task
+init_from_pretrain_model: ""
+# path of trained parameter, to make prediction
+init_from_params: "trained_params/step_100000/"
+# the directory for saving model
+save_model: "trained_models"
+# the directory for saving inference model.
+inference_model_dir: "infer_model"
+# Set seed for CE or debug
+random_seed: None
+# The pattern to match training data files.
+training_file: "wmt16_ende_data_bpe/train.tok.clean.bpe.32000.en-de"
+# The pattern to match validation data files.
+validation_file: "wmt16_ende_data_bpe/newstest2014.tok.bpe.32000.en-de"
+# The pattern to match test data files.
+predict_file: "wmt16_ende_data_bpe/newstest2016.tok.bpe.32000.en-de"
+# The file to output the translation results of predict_file to.
+output_file: "predict.txt"
+# The path of vocabulary file of source language.
+src_vocab_fpath: "wmt16_ende_data_bpe/vocab_all.bpe.32000"
+# The path of vocabulary file of target language.
+trg_vocab_fpath: "wmt16_ende_data_bpe/vocab_all.bpe.32000"
+# The , and tokens in the dictionary.
+special_token: ["", "", ""]
+
+# whether to use cuda
+use_cuda: True
+
+# args for reader, see reader.py for details
+token_delimiter: " "
+use_token_batch: True
+pool_size: 200000
+sort_type: "pool"
+shuffle: True
+shuffle_batch: True
+batch_size: 4096
+
+# Hyparams for training:
+# the number of epoches for training
+epoch: 30
+# the hyper parameters for Adam optimizer.
+# This static learning_rate will be multiplied to the LearningRateScheduler
+# derived learning rate the to get the final learning rate.
+learning_rate: 2.0
+beta1: 0.9
+beta2: 0.997
+eps: 1e-9
+# the parameters for learning rate scheduling.
+warmup_steps: 8000
+# the weight used to mix up the ground-truth distribution and the fixed
+# uniform distribution in label smoothing when training.
+# Set this as zero if label smoothing is not wanted.
+label_smooth_eps: 0.1
+
+# Hyparams for generation:
+# the parameters for beam search.
+beam_size: 5
+max_out_len: 256
+# the number of decoded sentences to output.
+n_best: 1
+
+# Hyparams for model:
+# These following five vocabularies related configurations will be set
+# automatically according to the passed vocabulary path and special tokens.
+# size of source word dictionary.
+src_vocab_size: 10000
+# size of target word dictionay
+trg_vocab_size: 10000
+# index for token
+bos_idx: 0
+# index for token
+eos_idx: 1
+# index for token
+unk_idx: 2
+# max length of sequences deciding the size of position encoding table.
+max_length: 256
+# the dimension for word embeddings, which is also the last dimension of
+# the input and output of multi-head attention, position-wise feed-forward
+# networks, encoder and decoder.
+d_model: 512
+# size of the hidden layer in position-wise feed-forward networks.
+d_inner_hid: 2048
+# the dimension that keys are projected to for dot-product attention.
+d_key: 64
+# the dimension that values are projected to for dot-product attention.
+d_value: 64
+# number of head used in multi-head attention.
+n_head: 8
+# number of sub-layers to be stacked in the encoder and decoder.
+n_layer: 6
+# dropout rates of different modules.
+prepostprocess_dropout: 0.1
+attention_dropout: 0.1
+relu_dropout: 0.1
+# to process before each sub-layer
+preprocess_cmd: "n" # layer normalization
+# to process after each sub-layer
+postprocess_cmd: "da" # dropout + residual connection
+# the flag indicating whether to share embedding and softmax weights.
+# vocabularies in source and target should be same for weight sharing.
+weight_sharing: True
diff --git a/transformer/utils/__init__.py b/transformer/utils/__init__.py
new file mode 100644
index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
diff --git a/transformer/utils/check.py b/transformer/utils/check.py
new file mode 100644
index 0000000000000000000000000000000000000000..305fa3705f5c313569986cbdb15c8afeda5a79c1
--- /dev/null
+++ b/transformer/utils/check.py
@@ -0,0 +1,61 @@
+# Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import sys
+
+import paddle.fluid as fluid
+
+import logging
+logger = logging.getLogger(__name__)
+
+__all__ = ['check_gpu', 'check_version']
+
+
+def check_gpu(use_gpu):
+ """
+ Log error and exit when set use_gpu=true in paddlepaddle
+ cpu version.
+ """
+ err = "Config use_gpu cannot be set as true while you are " \
+ "using paddlepaddle cpu version ! \nPlease try: \n" \
+ "\t1. Install paddlepaddle-gpu to run model on GPU \n" \
+ "\t2. Set use_gpu as false in config file to run " \
+ "model on CPU"
+
+ try:
+ if use_gpu and not fluid.is_compiled_with_cuda():
+ logger.error(err)
+ sys.exit(1)
+ except Exception as e:
+ pass
+
+
+def check_version():
+ """
+ Log error and exit when the installed version of paddlepaddle is
+ not satisfied.
+ """
+ err = "PaddlePaddle version 1.6 or higher is required, " \
+ "or a suitable develop version is satisfied as well. \n" \
+ "Please make sure the version is good with your code." \
+
+ try:
+ fluid.require_version('1.6.0')
+ except Exception as e:
+ logger.error(err)
+ sys.exit(1)
diff --git a/transformer/utils/configure.py b/transformer/utils/configure.py
new file mode 100644
index 0000000000000000000000000000000000000000..67e601282fee572518435eaed38a4ed8e26fc5f9
--- /dev/null
+++ b/transformer/utils/configure.py
@@ -0,0 +1,350 @@
+# Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import os
+import sys
+import argparse
+import json
+import yaml
+import six
+import logging
+
+logging_only_message = "%(message)s"
+logging_details = "%(asctime)s.%(msecs)03d %(levelname)s %(module)s - %(funcName)s: %(message)s"
+
+
+class JsonConfig(object):
+ """
+ A high-level api for handling json configure file.
+ """
+
+ def __init__(self, config_path):
+ self._config_dict = self._parse(config_path)
+
+ def _parse(self, config_path):
+ try:
+ with open(config_path) as json_file:
+ config_dict = json.load(json_file)
+ except:
+ raise IOError("Error in parsing bert model config file '%s'" %
+ config_path)
+ else:
+ return config_dict
+
+ def __getitem__(self, key):
+ return self._config_dict[key]
+
+ def print_config(self):
+ for arg, value in sorted(six.iteritems(self._config_dict)):
+ print('%s: %s' % (arg, value))
+ print('------------------------------------------------')
+
+
+class ArgumentGroup(object):
+ def __init__(self, parser, title, des):
+ self._group = parser.add_argument_group(title=title, description=des)
+
+ def add_arg(self, name, type, default, help, **kwargs):
+ type = str2bool if type == bool else type
+ self._group.add_argument(
+ "--" + name,
+ default=default,
+ type=type,
+ help=help + ' Default: %(default)s.',
+ **kwargs)
+
+
+class ArgConfig(object):
+ """
+ A high-level api for handling argument configs.
+ """
+
+ def __init__(self):
+ parser = argparse.ArgumentParser()
+
+ train_g = ArgumentGroup(parser, "training", "training options.")
+ train_g.add_arg("epoch", int, 3, "Number of epoches for fine-tuning.")
+ train_g.add_arg("learning_rate", float, 5e-5,
+ "Learning rate used to train with warmup.")
+ train_g.add_arg(
+ "lr_scheduler",
+ str,
+ "linear_warmup_decay",
+ "scheduler of learning rate.",
+ choices=['linear_warmup_decay', 'noam_decay'])
+ train_g.add_arg("weight_decay", float, 0.01,
+ "Weight decay rate for L2 regularizer.")
+ train_g.add_arg(
+ "warmup_proportion", float, 0.1,
+ "Proportion of training steps to perform linear learning rate warmup for."
+ )
+ train_g.add_arg("save_steps", int, 1000,
+ "The steps interval to save checkpoints.")
+ train_g.add_arg("use_fp16", bool, False,
+ "Whether to use fp16 mixed precision training.")
+ train_g.add_arg(
+ "loss_scaling", float, 1.0,
+ "Loss scaling factor for mixed precision training, only valid when use_fp16 is enabled."
+ )
+ train_g.add_arg("pred_dir", str, None,
+ "Path to save the prediction results")
+
+ log_g = ArgumentGroup(parser, "logging", "logging related.")
+ log_g.add_arg("skip_steps", int, 10,
+ "The steps interval to print loss.")
+ log_g.add_arg("verbose", bool, False, "Whether to output verbose log.")
+
+ run_type_g = ArgumentGroup(parser, "run_type", "running type options.")
+ run_type_g.add_arg("use_cuda", bool, True,
+ "If set, use GPU for training.")
+ run_type_g.add_arg(
+ "use_fast_executor", bool, False,
+ "If set, use fast parallel executor (in experiment).")
+ run_type_g.add_arg(
+ "num_iteration_per_drop_scope", int, 1,
+ "Ihe iteration intervals to clean up temporary variables.")
+ run_type_g.add_arg("do_train", bool, True,
+ "Whether to perform training.")
+ run_type_g.add_arg("do_predict", bool, True,
+ "Whether to perform prediction.")
+
+ custom_g = ArgumentGroup(parser, "customize", "customized options.")
+
+ self.custom_g = custom_g
+
+ self.parser = parser
+
+ def add_arg(self, name, dtype, default, descrip):
+ self.custom_g.add_arg(name, dtype, default, descrip)
+
+ def build_conf(self):
+ return self.parser.parse_args()
+
+
+def str2bool(v):
+ # because argparse does not support to parse "true, False" as python
+ # boolean directly
+ return v.lower() in ("true", "t", "1")
+
+
+def print_arguments(args, log=None):
+ if not log:
+ print('----------- Configuration Arguments -----------')
+ for arg, value in sorted(six.iteritems(vars(args))):
+ print('%s: %s' % (arg, value))
+ print('------------------------------------------------')
+ else:
+ log.info('----------- Configuration Arguments -----------')
+ for arg, value in sorted(six.iteritems(vars(args))):
+ log.info('%s: %s' % (arg, value))
+ log.info('------------------------------------------------')
+
+
+class PDConfig(object):
+ """
+ A high-level API for managing configuration files in PaddlePaddle.
+ Can jointly work with command-line-arugment, json files and yaml files.
+ """
+
+ def __init__(self, json_file="", yaml_file="", fuse_args=True):
+ """
+ Init funciton for PDConfig.
+ json_file: the path to the json configure file.
+ yaml_file: the path to the yaml configure file.
+ fuse_args: if fuse the json/yaml configs with argparse.
+ """
+ assert isinstance(json_file, str)
+ assert isinstance(yaml_file, str)
+
+ if json_file != "" and yaml_file != "":
+ raise Warning(
+ "json_file and yaml_file can not co-exist for now. please only use one configure file type."
+ )
+ return
+
+ self.args = None
+ self.arg_config = {}
+ self.json_config = {}
+ self.yaml_config = {}
+
+ parser = argparse.ArgumentParser()
+
+ self.default_g = ArgumentGroup(parser, "default", "default options.")
+ self.yaml_g = ArgumentGroup(parser, "yaml", "options from yaml.")
+ self.json_g = ArgumentGroup(parser, "json", "options from json.")
+ self.com_g = ArgumentGroup(parser, "custom", "customized options.")
+
+ self.default_g.add_arg("do_train", bool, False,
+ "Whether to perform training.")
+ self.default_g.add_arg("do_predict", bool, False,
+ "Whether to perform predicting.")
+ self.default_g.add_arg("do_eval", bool, False,
+ "Whether to perform evaluating.")
+ self.default_g.add_arg("do_save_inference_model", bool, False,
+ "Whether to perform model saving for inference.")
+
+ # NOTE: args for profiler
+ self.default_g.add_arg("is_profiler", int, 0, "the switch of profiler tools. (used for benchmark)")
+ self.default_g.add_arg("profiler_path", str, './', "the profiler output file path. (used for benchmark)")
+ self.default_g.add_arg("max_iter", int, 0, "the max train batch num.(used for benchmark)")
+
+ self.parser = parser
+
+ if json_file != "":
+ self.load_json(json_file, fuse_args=fuse_args)
+
+ if yaml_file:
+ self.load_yaml(yaml_file, fuse_args=fuse_args)
+
+ def load_json(self, file_path, fuse_args=True):
+
+ if not os.path.exists(file_path):
+ raise Warning("the json file %s does not exist." % file_path)
+ return
+
+ with open(file_path, "r") as fin:
+ self.json_config = json.loads(fin.read())
+ fin.close()
+
+ if fuse_args:
+ for name in self.json_config:
+ if isinstance(self.json_config[name], list):
+ self.json_g.add_arg(
+ name,
+ type(self.json_config[name][0]),
+ self.json_config[name],
+ "This is from %s" % file_path,
+ nargs=len(self.json_config[name]))
+ continue
+ if not isinstance(self.json_config[name], int) \
+ and not isinstance(self.json_config[name], float) \
+ and not isinstance(self.json_config[name], str) \
+ and not isinstance(self.json_config[name], bool):
+
+ continue
+
+ self.json_g.add_arg(name,
+ type(self.json_config[name]),
+ self.json_config[name],
+ "This is from %s" % file_path)
+
+ def load_yaml(self, file_path, fuse_args=True):
+
+ if not os.path.exists(file_path):
+ raise Warning("the yaml file %s does not exist." % file_path)
+ return
+
+ with open(file_path, "r") as fin:
+ self.yaml_config = yaml.load(fin, Loader=yaml.SafeLoader)
+ fin.close()
+
+ if fuse_args:
+ for name in self.yaml_config:
+ if isinstance(self.yaml_config[name], list):
+ self.yaml_g.add_arg(
+ name,
+ type(self.yaml_config[name][0]),
+ self.yaml_config[name],
+ "This is from %s" % file_path,
+ nargs=len(self.yaml_config[name]))
+ continue
+
+ if not isinstance(self.yaml_config[name], int) \
+ and not isinstance(self.yaml_config[name], float) \
+ and not isinstance(self.yaml_config[name], str) \
+ and not isinstance(self.yaml_config[name], bool):
+
+ continue
+
+ self.yaml_g.add_arg(name,
+ type(self.yaml_config[name]),
+ self.yaml_config[name],
+ "This is from %s" % file_path)
+
+ def build(self):
+ self.args = self.parser.parse_args()
+ self.arg_config = vars(self.args)
+
+ def __add__(self, new_arg):
+ assert isinstance(new_arg, list) or isinstance(new_arg, tuple)
+ assert len(new_arg) >= 3
+ assert self.args is None
+
+ name = new_arg[0]
+ dtype = new_arg[1]
+ dvalue = new_arg[2]
+ desc = new_arg[3] if len(
+ new_arg) == 4 else "Description is not provided."
+
+ self.com_g.add_arg(name, dtype, dvalue, desc)
+
+ return self
+
+ def __getattr__(self, name):
+ if name in self.arg_config:
+ return self.arg_config[name]
+
+ if name in self.json_config:
+ return self.json_config[name]
+
+ if name in self.yaml_config:
+ return self.yaml_config[name]
+
+ raise Warning("The argument %s is not defined." % name)
+
+ def Print(self):
+
+ print("-" * 70)
+ for name in self.arg_config:
+ print("%s:\t\t\t\t%s" % (str(name), str(self.arg_config[name])))
+
+ for name in self.json_config:
+ if name not in self.arg_config:
+ print("%s:\t\t\t\t%s" %
+ (str(name), str(self.json_config[name])))
+
+ for name in self.yaml_config:
+ if name not in self.arg_config:
+ print("%s:\t\t\t\t%s" %
+ (str(name), str(self.yaml_config[name])))
+
+ print("-" * 70)
+
+
+if __name__ == "__main__":
+ """
+ pd_config = PDConfig(json_file = "./test/bert_config.json")
+ pd_config.build()
+
+ print(pd_config.do_train)
+ print(pd_config.hidden_size)
+
+ pd_config = PDConfig(yaml_file = "./test/bert_config.yaml")
+ pd_config.build()
+
+ print(pd_config.do_train)
+ print(pd_config.hidden_size)
+ """
+
+ pd_config = PDConfig(yaml_file="./test/bert_config.yaml")
+ pd_config += ("my_age", int, 18, "I am forever 18.")
+ pd_config.build()
+
+ print(pd_config.do_train)
+ print(pd_config.hidden_size)
+ print(pd_config.my_age)
diff --git a/yolov3.py b/yolov3.py
deleted file mode 100644
index 6c609f24dce60293ee42a599324d595a6875a0f6..0000000000000000000000000000000000000000
--- a/yolov3.py
+++ /dev/null
@@ -1,568 +0,0 @@
-# Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from __future__ import division
-from __future__ import print_function
-
-import argparse
-import contextlib
-import os
-import random
-import time
-
-from functools import partial
-
-import cv2
-import numpy as np
-from pycocotools.coco import COCO
-
-import paddle
-import paddle.fluid as fluid
-from paddle.fluid.dygraph.nn import Conv2D
-from paddle.fluid.param_attr import ParamAttr
-from paddle.fluid.regularizer import L2Decay
-
-from model import Model, Loss, Input
-from resnet import ResNet, ConvBNLayer
-
-import logging
-FORMAT = '%(asctime)s-%(levelname)s: %(message)s'
-logging.basicConfig(level=logging.INFO, format=FORMAT)
-logger = logging.getLogger(__name__)
-
-
-# XXX transfer learning
-class ResNetBackBone(ResNet):
- def __init__(self, depth=50):
- super(ResNetBackBone, self).__init__(depth=depth)
- delattr(self, 'fc')
-
- def forward(self, inputs):
- x = self.conv(inputs)
- x = self.pool(x)
- outputs = []
- for layer in self.layers:
- x = layer(x)
- outputs.append(x)
- return outputs
-
-
-class YoloDetectionBlock(fluid.dygraph.Layer):
- def __init__(self, num_channels, num_filters):
- super(YoloDetectionBlock, self).__init__()
-
- assert num_filters % 2 == 0, \
- "num_filters {} cannot be divided by 2".format(num_filters)
-
- self.conv0 = ConvBNLayer(
- num_channels=num_channels,
- num_filters=num_filters,
- filter_size=1,
- act='leaky_relu')
- self.conv1 = ConvBNLayer(
- num_channels=num_filters,
- num_filters=num_filters * 2,
- filter_size=3,
- act='leaky_relu')
- self.conv2 = ConvBNLayer(
- num_channels=num_filters * 2,
- num_filters=num_filters,
- filter_size=1,
- act='leaky_relu')
- self.conv3 = ConvBNLayer(
- num_channels=num_filters,
- num_filters=num_filters * 2,
- filter_size=3,
- act='leaky_relu')
- self.route = ConvBNLayer(
- num_channels=num_filters * 2,
- num_filters=num_filters,
- filter_size=1,
- act='leaky_relu')
- self.tip = ConvBNLayer(
- num_channels=num_filters,
- num_filters=num_filters * 2,
- filter_size=3,
- act='leaky_relu')
-
- def forward(self, inputs):
- out = self.conv0(inputs)
- out = self.conv1(out)
- out = self.conv2(out)
- out = self.conv3(out)
- route = self.route(out)
- tip = self.tip(route)
- return route, tip
-
-
-class YOLOv3(Model):
- def __init__(self, num_classes=80):
- super(YOLOv3, self).__init__()
- self.num_classes = num_classes
- self.anchors = [10, 13, 16, 30, 33, 23, 30, 61, 62, 45,
- 59, 119, 116, 90, 156, 198, 373, 326]
- self.anchor_masks = [[6, 7, 8], [3, 4, 5], [0, 1, 2]]
- self.valid_thresh = 0.005
- self.nms_thresh = 0.45
- self.nms_topk = 400
- self.nms_posk = 100
- self.draw_thresh = 0.5
-
- self.backbone = ResNetBackBone()
- self.block_outputs = []
- self.yolo_blocks = []
- self.route_blocks = []
-
- for idx, num_chan in enumerate([2048, 1280, 640]):
- yolo_block = self.add_sublayer(
- "detecton_block_{}".format(idx),
- YoloDetectionBlock(num_chan, num_filters=512 // (2**idx)))
- self.yolo_blocks.append(yolo_block)
-
- num_filters = len(self.anchor_masks[idx]) * (self.num_classes + 5)
-
- block_out = self.add_sublayer(
- "block_out_{}".format(idx),
- Conv2D(num_channels=1024 // (2**idx),
- num_filters=num_filters,
- filter_size=1,
- param_attr=ParamAttr(
- initializer=fluid.initializer.Normal(0., 0.02)),
- bias_attr=ParamAttr(
- initializer=fluid.initializer.Constant(0.0),
- regularizer=L2Decay(0.))))
- self.block_outputs.append(block_out)
- if idx < 2:
- route = self.add_sublayer(
- "route_{}".format(idx),
- ConvBNLayer(num_channels=512 // (2**idx),
- num_filters=256 // (2**idx),
- filter_size=1,
- act='leaky_relu'))
- self.route_blocks.append(route)
-
- def forward(self, inputs, img_info):
- outputs = []
- boxes = []
- scores = []
- downsample = 32
-
- feats = self.backbone(inputs)
- feats = feats[::-1][:len(self.anchor_masks)]
- route = None
- for idx, feat in enumerate(feats):
- if idx > 0:
- feat = fluid.layers.concat(input=[route, feat], axis=1)
- route, tip = self.yolo_blocks[idx](feat)
- block_out = self.block_outputs[idx](tip)
- outputs.append(block_out)
-
- if idx < 2:
- route = self.route_blocks[idx](route)
- route = fluid.layers.resize_nearest(route, scale=2)
-
- if self.mode == 'test':
- anchor_mask = self.anchor_masks[idx]
- mask_anchors = []
- for m in anchor_mask:
- mask_anchors.append(self.anchors[2 * m])
- mask_anchors.append(self.anchors[2 * m + 1])
- img_shape = fluid.layers.slice(img_info, axes=[1], starts=[1], ends=[3])
- img_id = fluid.layers.slice(img_info, axes=[1], starts=[0], ends=[1])
- b, s = fluid.layers.yolo_box(
- x=block_out,
- img_size=img_shape,
- anchors=mask_anchors,
- class_num=self.num_classes,
- conf_thresh=self.valid_thresh,
- downsample_ratio=downsample)
-
- boxes.append(b)
- scores.append(fluid.layers.transpose(s, perm=[0, 2, 1]))
-
- downsample //= 2
-
- if self.mode != 'test':
- return outputs
-
- return [img_id, fluid.layers.multiclass_nms(
- bboxes=fluid.layers.concat(boxes, axis=1),
- scores=fluid.layers.concat(scores, axis=2),
- score_threshold=self.valid_thresh,
- nms_top_k=self.nms_topk,
- keep_top_k=self.nms_posk,
- nms_threshold=self.nms_thresh,
- background_label=-1)]
-
-
-class YoloLoss(Loss):
- def __init__(self, num_classes=80):
- super(YoloLoss, self).__init__()
- self.num_classes = num_classes
- self.ignore_thresh = 0.7
- self.anchors = [10, 13, 16, 30, 33, 23, 30, 61, 62, 45,
- 59, 119, 116, 90, 156, 198, 373, 326]
- self.anchor_masks = [[6, 7, 8], [3, 4, 5], [0, 1, 2]]
-
- def forward(self, outputs, labels):
- downsample = 32
- gt_box, gt_label, gt_score = labels
- losses = []
-
- for idx, out in enumerate(outputs):
- anchor_mask = self.anchor_masks[idx]
- loss = fluid.layers.yolov3_loss(
- x=out,
- gt_box=gt_box,
- gt_label=gt_label,
- gt_score=gt_score,
- anchor_mask=anchor_mask,
- downsample_ratio=downsample,
- anchors=self.anchors,
- class_num=self.num_classes,
- ignore_thresh=self.ignore_thresh,
- use_label_smooth=True)
- loss = fluid.layers.reduce_mean(loss)
- losses.append(loss)
- downsample //= 2
- return losses
-
-
-def make_optimizer(parameter_list=None):
- base_lr = FLAGS.lr
- warm_up_iter = 4000
- momentum = 0.9
- weight_decay = 5e-4
- boundaries = [400000, 450000]
- values = [base_lr * (0.1 ** i) for i in range(len(boundaries) + 1)]
- learning_rate = fluid.layers.piecewise_decay(
- boundaries=boundaries,
- values=values)
- learning_rate = fluid.layers.linear_lr_warmup(
- learning_rate=learning_rate,
- warmup_steps=warm_up_iter,
- start_lr=0.0,
- end_lr=base_lr)
- optimizer = fluid.optimizer.Momentum(
- learning_rate=learning_rate,
- regularization=fluid.regularizer.L2Decay(weight_decay),
- momentum=momentum,
- parameter_list=parameter_list)
- return optimizer
-
-
-def _iou_matrix(a, b):
- tl_i = np.maximum(a[:, np.newaxis, :2], b[:, :2])
- br_i = np.minimum(a[:, np.newaxis, 2:], b[:, 2:])
- area_i = np.prod(br_i - tl_i, axis=2) * (tl_i < br_i).all(axis=2)
- area_a = np.prod(a[:, 2:] - a[:, :2], axis=1)
- area_b = np.prod(b[:, 2:] - b[:, :2], axis=1)
- area_o = (area_a[:, np.newaxis] + area_b - area_i)
- return area_i / (area_o + 1e-10)
-
-
-def _crop_box_with_center_constraint(box, crop):
- cropped_box = box.copy()
- cropped_box[:, :2] = np.maximum(box[:, :2], crop[:2])
- cropped_box[:, 2:] = np.minimum(box[:, 2:], crop[2:])
- cropped_box[:, :2] -= crop[:2]
- cropped_box[:, 2:] -= crop[:2]
- centers = (box[:, :2] + box[:, 2:]) / 2
- valid = np.logical_and(
- crop[:2] <= centers, centers < crop[2:]).all(axis=1)
- valid = np.logical_and(
- valid, (cropped_box[:, :2] < cropped_box[:, 2:]).all(axis=1))
- return cropped_box, np.where(valid)[0]
-
-
-def random_crop(inputs):
- aspect_ratios = [.5, 2.]
- thresholds = [.0, .1, .3, .5, .7, .9]
- scaling = [.3, 1.]
-
- img, img_ids, gt_box, gt_label = inputs
- h, w = img.shape[:2]
-
- if len(gt_box) == 0:
- return inputs
-
- np.random.shuffle(thresholds)
- for thresh in thresholds:
- found = False
- for i in range(50):
- scale = np.random.uniform(*scaling)
- min_ar, max_ar = aspect_ratios
- ar = np.random.uniform(max(min_ar, scale**2),
- min(max_ar, scale**-2))
- crop_h = int(h * scale / np.sqrt(ar))
- crop_w = int(w * scale * np.sqrt(ar))
- crop_y = np.random.randint(0, h - crop_h)
- crop_x = np.random.randint(0, w - crop_w)
- crop_box = [crop_x, crop_y, crop_x + crop_w, crop_y + crop_h]
- iou = _iou_matrix(gt_box, np.array([crop_box], dtype=np.float32))
- if iou.max() < thresh:
- continue
-
- cropped_box, valid_ids = _crop_box_with_center_constraint(
- gt_box, np.array(crop_box, dtype=np.float32))
- if valid_ids.size > 0:
- found = True
- break
-
- if found:
- x1, y1, x2, y2 = crop_box
- img = img[y1:y2, x1:x2, :]
- gt_box = np.take(cropped_box, valid_ids, axis=0)
- gt_label = np.take(gt_label, valid_ids, axis=0)
- return img, img_ids, gt_box, gt_label
-
- return inputs
-
-
-# XXX mix up, color distort and random expand are skipped for simplicity
-def sample_transform(inputs, mode='train', num_max_boxes=50):
- if mode == 'train':
- img, img_id, gt_box, gt_label = random_crop(inputs)
- else:
- img, img_id, gt_box, gt_label = inputs
-
- h, w = img.shape[:2]
- # random flip
- if mode == 'train' and np.random.uniform(0., 1.) > .5:
- img = img[:, ::-1, :]
- if len(gt_box) > 0:
- swap = gt_box.copy()
- gt_box[:, 0] = w - swap[:, 2] - 1
- gt_box[:, 2] = w - swap[:, 0] - 1
-
- if len(gt_label) == 0:
- gt_box = np.zeros([num_max_boxes, 4], dtype=np.float32)
- gt_label = np.zeros([num_max_boxes], dtype=np.int32)
- return img, gt_box, gt_label
-
- gt_box = gt_box[:num_max_boxes, :]
- gt_label = gt_label[:num_max_boxes, 0]
- # normalize boxes
- gt_box /= np.array([w, h] * 2, dtype=np.float32)
- gt_box[:, 2:] = gt_box[:, 2:] - gt_box[:, :2]
- gt_box[:, :2] = gt_box[:, :2] + gt_box[:, 2:] / 2.
-
- pad = num_max_boxes - gt_label.size
- gt_box = np.pad(gt_box, ((0, pad), (0, 0)), mode='constant')
- gt_label = np.pad(gt_label, ((0, pad)), mode='constant')
-
- return img, img_id, gt_box, gt_label
-
-
-def batch_transform(batch, mode='train'):
- if mode == 'train':
- d = np.random.choice(
- [320, 352, 384, 416, 448, 480, 512, 544, 576, 608])
- interp = np.random.choice(range(5))
- else:
- d = 608
- interp = cv2.INTER_CUBIC
- # transpose batch
- imgs, img_ids, gt_boxes, gt_labels = list(zip(*batch))
- img_shapes = np.array([[im.shape[0], im.shape[1]] for im in imgs]).astype('int32')
- imgs = np.array([cv2.resize(
- img, (d, d), interpolation=interp) for img in imgs])
-
- # transpose, permute and normalize
- imgs = imgs.astype(np.float32)[..., ::-1]
- mean = np.array([123.675, 116.28, 103.53], dtype=np.float32)
- std = np.array([58.395, 57.120, 57.375], dtype=np.float32)
- invstd = 1. / std
- imgs -= mean
- imgs *= invstd
- imgs = imgs.transpose((0, 3, 1, 2))
-
- img_ids = np.array(img_ids)
- img_info = np.concatenate([img_ids, img_shapes], axis=1)
- gt_boxes = np.array(gt_boxes)
- gt_labels = np.array(gt_labels)
- # XXX since mix up is not used, scores are all ones
- gt_scores = np.ones_like(gt_labels, dtype=np.float32)
- return [imgs, img_info], [gt_boxes, gt_labels, gt_scores]
-
-
-def coco2017(root_dir, mode='train'):
- json_path = os.path.join(
- root_dir, 'annotations/instances_{}2017.json'.format(mode))
- coco = COCO(json_path)
- img_ids = coco.getImgIds()
- imgs = coco.loadImgs(img_ids)
- class_map = {v: i + 1 for i, v in enumerate(coco.getCatIds())}
- samples = []
-
- for img in imgs:
- img_path = os.path.join(
- root_dir, '{}2017'.format(mode), img['file_name'])
- file_path = img_path
- width = img['width']
- height = img['height']
- ann_ids = coco.getAnnIds(imgIds=img['id'], iscrowd=False)
- anns = coco.loadAnns(ann_ids)
-
- gt_box = []
- gt_label = []
-
- for ann in anns:
- x1, y1, w, h = ann['bbox']
- x2 = x1 + w - 1
- y2 = y1 + h - 1
- x1 = np.clip(x1, 0, width - 1)
- x2 = np.clip(x2, 0, width - 1)
- y1 = np.clip(y1, 0, height - 1)
- y2 = np.clip(y2, 0, height - 1)
- if ann['area'] <= 0 or x2 < x1 or y2 < y1:
- continue
- gt_label.append(ann['category_id'])
- gt_box.append([x1, y1, x2, y2])
-
- gt_box = np.array(gt_box, dtype=np.float32)
- gt_label = np.array([class_map[cls] for cls in gt_label],
- dtype=np.int32)[:, np.newaxis]
- im_id = np.array([img['id']], dtype=np.int32)
-
- if gt_label.size == 0 and not mode == 'train':
- continue
- samples.append((file_path, im_id.copy(), gt_box.copy(), gt_label.copy()))
-
- def iterator():
- if mode == 'train':
- np.random.shuffle(samples)
- for file_path, im_id, gt_box, gt_label in samples:
- img = cv2.imread(file_path)
- yield img, im_id, gt_box, gt_label
-
- return iterator
-
-
-# XXX coco metrics not included for simplicity
-def run(model, loader, mode='train'):
- total_loss = 0.
- total_time = 0.
- device_ids = list(range(FLAGS.num_devices))
- start = time.time()
-
- for idx, batch in enumerate(loader()):
- losses = getattr(model, mode)(batch[0], batch[1])
-
- total_loss += np.sum(losses)
- if idx > 1: # skip first two steps
- total_time += time.time() - start
- if idx % 10 == 0:
- logger.info("{:04d}: loss {:0.3f} time: {:0.3f}".format(
- idx, total_loss / (idx + 1), total_time / max(1, (idx - 1))))
- start = time.time()
-
-
-def main():
- @contextlib.contextmanager
- def null_guard():
- yield
-
- epoch = FLAGS.epoch
- batch_size = FLAGS.batch_size
- guard = fluid.dygraph.guard() if FLAGS.dynamic else null_guard()
-
- train_loader = fluid.io.xmap_readers(
- batch_transform,
- paddle.batch(
- fluid.io.xmap_readers(
- sample_transform,
- coco2017(FLAGS.data, 'train'),
- process_num=8,
- buffer_size=4 * batch_size),
- batch_size=batch_size,
- drop_last=True),
- process_num=2, buffer_size=4)
-
- val_sample_transform = partial(sample_transform, mode='val')
- val_batch_transform = partial(batch_transform, mode='val')
-
- val_loader = fluid.io.xmap_readers(
- val_batch_transform,
- paddle.batch(
- fluid.io.xmap_readers(
- val_sample_transform,
- coco2017(FLAGS.data, 'val'),
- process_num=8,
- buffer_size=4 * batch_size),
- batch_size=1),
- process_num=2, buffer_size=4)
-
- if not os.path.exists('yolo_checkpoints'):
- os.mkdir('yolo_checkpoints')
-
- with guard:
- NUM_CLASSES = 7
- NUM_MAX_BOXES = 50
- model = YOLOv3(num_classes=NUM_CLASSES)
- # XXX transfer learning
- if FLAGS.pretrain_weights is not None:
- model.backbone.load(FLAGS.pretrain_weights)
- if FLAGS.weights is not None:
- model.load(FLAGS.weights)
- optim = make_optimizer(parameter_list=model.parameters())
- anno_path = os.path.join(FLAGS.data, 'annotations', 'instances_val2017.json')
- inputs = [Input([None, 3, None, None], 'float32', name='image'),
- Input([None, 3], 'int32', name='img_info')]
- labels = [Input([None, NUM_MAX_BOXES, 4], 'float32', name='gt_bbox'),
- Input([None, NUM_MAX_BOXES], 'int32', name='gt_label'),
- Input([None, NUM_MAX_BOXES], 'float32', name='gt_score')]
- model.prepare(optim,
- YoloLoss(num_classes=NUM_CLASSES),
- # For YOLOv3, output variable in train/eval is different,
- # which is not supported by metric, add by callback later?
- # metrics=COCOMetric(anno_path, with_background=False)
- inputs=inputs,
- labels = labels)
-
- for e in range(epoch):
- logger.info("======== train epoch {} ========".format(e))
- run(model, train_loader)
- model.save('yolo_checkpoints/{:02d}'.format(e))
- logger.info("======== eval epoch {} ========".format(e))
- run(model, val_loader, mode='eval')
- # should be called in fit()
- for metric in model._metrics:
- metric.accumulate()
- metric.reset()
-
-
-if __name__ == '__main__':
- parser = argparse.ArgumentParser("Yolov3 Training on COCO")
- parser.add_argument('data', metavar='DIR', help='path to COCO dataset')
- parser.add_argument(
- "-d", "--dynamic", action='store_true', help="enable dygraph mode")
- parser.add_argument(
- "-e", "--epoch", default=300, type=int, help="number of epoch")
- parser.add_argument(
- '--lr', '--learning-rate', default=0.001, type=float, metavar='LR',
- help='initial learning rate')
- parser.add_argument(
- "-b", "--batch_size", default=64, type=int, help="batch size")
- parser.add_argument(
- "-n", "--num_devices", default=8, type=int, help="number of devices")
- parser.add_argument(
- "-p", "--pretrain_weights", default=None, type=str,
- help="path to pretrained weights")
- parser.add_argument(
- "-w", "--weights", default=None, type=str,
- help="path to model weights")
- FLAGS = parser.parse_args()
- assert FLAGS.data, "error: must provide data path"
- main()
diff --git a/yolov3/README.md b/yolov3/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..88e153509c4e14c21c43dcc1d69fece7998616e1
--- /dev/null
+++ b/yolov3/README.md
@@ -0,0 +1,203 @@
+# YOLOv3 目标检测模型
+
+---
+
+## 内容
+
+- [模型简介](#模型简介)
+- [快速开始](#快速开始)
+- [参考论文](#参考论文)
+
+
+## 模型简介
+
+[YOLOv3](https://arxiv.org/abs/1804.02767) 是由 [Joseph Redmon](https://arxiv.org/search/cs?searchtype=author&query=Redmon%2C+J) 和 [Ali Farhadi](https://arxiv.org/search/cs?searchtype=author&query=Farhadi%2C+A) 提出的单阶段检测器, 该检测器与达到同样精度的传统目标检测方法相比，推断速度能达到接近两倍.
+
+传统目标检测方法通过两阶段检测，第一阶段生成预选框，第二阶段对预选框进行分类和位置坐标的调整，而YOLO将目标检测看做是对框位置和类别概率的一个单阶段回归问题，使得YOLO能达到近两倍的检测速度。而YOLOv3在YOLO的基础上引入的多尺度预测，使得YOLOv3网络对于小物体的检测精度大幅提高。
+
+[YOLOv3](https://arxiv.org/abs/1804.02767) 是一阶段End2End的目标检测器。其目标检测原理如下图所示:
+
+
+YOLOv3检测原理
+
+
+YOLOv3将输入图像分成S\*S个格子，每个格子预测B个bounding box，每个bounding box预测内容包括: Location(x, y, w, h)、Confidence Score和C个类别的概率，因此YOLOv3输出层的channel数为B\*(5 + C)。YOLOv3的loss函数也有三部分组成：Location误差，Confidence误差和分类误差。
+
+YOLOv3的网络结构如下图所示:
+
+
+YOLOv3网络结构
+
+
+YOLOv3 的网络结构由基础特征提取网络、multi-scale特征融合层和输出层组成。
+
+1. 特征提取网络。YOLOv3使用 [DarkNet53](https://arxiv.org/abs/1612.08242)作为特征提取网络：DarkNet53 基本采用了全卷积网络，用步长为2的卷积操作替代了池化层，同时添加了 Residual 单元，避免在网络层数过深时发生梯度弥散。
+
+2. 特征融合层。为了解决之前YOLO版本对小目标不敏感的问题，YOLOv3采用了3个不同尺度的特征图来进行目标检测，分别为13\*13,26\*26,52\*52,用来检测大、中、小三种目标。特征融合层选取 DarkNet 产出的三种尺度特征图作为输入，借鉴了FPN(feature pyramid networks)的思想，通过一系列的卷积层和上采样对各尺度的特征图进行融合。
+
+3. 输出层。同样使用了全卷积结构，其中最后一个卷积层的卷积核个数是255：3\*(80+4+1)=255，3表示一个grid cell包含3个bounding box，4表示框的4个坐标信息，1表示Confidence Score，80表示COCO数据集中80个类别的概率。
+
+
+## 快速开始
+
+### 安装说明
+
+#### paddle安装
+
+ 本项目依赖于 PaddlePaddle 1.7及以上版本或适当的develop版本，请参考 [安装指南](http://www.paddlepaddle.org/#quick-start) 进行安装
+
+#### 代码下载及环境变量设置
+
+ 克隆代码库到本地，并设置`PYTHONPATH`环境变量
+
+ ```bash
+ git clone https://github.com/PaddlePaddle/hapi
+ cd hapi
+ export PYTHONPATH=$PYTHONPATH:`pwd`
+ cd tsm
+ ```
+
+#### 安装COCO-API
+
+ 训练前需要首先下载[COCO-API](https://github.com/cocodataset/cocoapi)：
+
+ ```bash
+ git clone https://github.com/cocodataset/cocoapi.git
+ cd cocoapi/PythonAPI
+ # if cython is not installed
+ pip install Cython
+ # Install into global site-packages
+ make install
+ # Alternatively, if you do not have permissions or prefer
+ # not to install the COCO API into global site-packages
+ python setup.py install --user
+ ```
+
+### 数据准备
+
+模型目前支持COCO数据集格式的数据读入和精度评估，我们同时提供了将转换为COCO数据集的格式的Pascal VOC数据集下载，可通过如下命令下载。
+
+ ```bash
+ python dataset/download_voc.py
+ ```
+
+数据目录结构如下：
+
+ ```
+ dataset/voc/
+ ├── annotations
+ │ ├── instances_train2017.json
+ │ ├── instances_val2017.json
+ | ...
+ ├── train2017
+ │ ├── 1013.jpg
+ │ ├── 1014.jpg
+ | ...
+ ├── val2017
+ │ ├── 2551.jpg
+ │ ├── 2552.jpg
+ | ...
+ ```
+
+### 模型训练
+
+数据准备完毕后，可使用`main.py`脚本启动训练和评估，如下脚本会自动每epoch交替进行训练和模型评估，并将checkpoint默认保存在`yolo_checkpoint`目录下。
+
+YOLOv3模型训练总batch_size为64训练，以下以使用4卡Tesla P40每卡batch_size为16训练介绍训练方式。对于静态图和动态图，多卡训练中`--batch_size`为每卡上的batch_size，即总batch_size为`--batch_size`乘以卡数。
+
+
+`main.py`脚本参数可通过如下命令查询
+
+```bash
+python main.py --help
+```
+
+#### 静态图训练
+
+使用如下方式进行多卡训练:
+
+```bash
+CUDA_VISIBLE_DEVICES=0,1,2,3 python main.py --data= --batch_size=16
+```
+
+#### 动态图训练
+
+动态图训练只需要在运行脚本时添加`-d`参数即可。
+
+使用如下方式进行多卡训练:
+
+```bash
+CUDA_VISIBLE_DEVICES=0,1,2,3 python main.py --data= --batch_size=16 -d
+```
+
+
+### 模型评估
+
+YOLOv3模型输出为LoDTensor，只支持使用batch_size为1进行评估，可通过如下两种方式进行模型评估。
+
+1. 自动下载Paddle发布的[YOLOv3-DarkNet53](https://paddlemodels.bj.bcebos.com/hapi/yolov3_darknet53.pdparams)权重评估
+
+```bash
+python main.py --data=dataset/voc --eval_only
+```
+
+2. 加载checkpoint进行精度评估
+
+```bash
+python main.py --data=dataset/voc --eval_only --weights=yolo_checkpoint/no_mixup/final
+```
+
+同样可以通过指定`-d`参数进行动态图模式的评估。
+
+#### 评估精度
+
+在10类小数据集下训练模型权重见[YOLOv3-DarkNet53](https://paddlemodels.bj.bcebos.com/hapi/yolov3_darknet53.pdparams)，评估精度如下：
+
+```bash
+Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.503
+Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.779
+Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.562
+Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.190
+Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.390
+Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.578
+Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.405
+Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.591
+Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.599
+Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.294
+Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.506
+Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.670
+```
+
+### 模型推断及可视化
+
+可通过如下两种方式进行模型推断。
+
+1. 自动下载Paddle发布的[YOLOv3-DarkNet53](https://paddlemodels.bj.bcebos.com/hapi/yolov3_darknet53.pdparams)权重评估
+
+```bash
+python infer.py --label_list=dataset/voc/label_list.txt --infer_image=image/dog.jpg
+```
+
+2. 加载checkpoint进行精度评估
+
+```bash
+python infer.py --label_list=dataset/voc/label_list.txt --infer_image=image/dog.jpg --weights=yolo_checkpoint/mo_mixup/final
+```
+
+推断结果可视化图像会保存于`--output`指定的文件夹下，默认保存于`./output`目录。
+
+模型推断会输出如下检测结果日志：
+
+```text
+2020-04-02 08:26:47,268-INFO: detect bicycle at [116.14993, 127.278336, 579.7716, 438.44214] score: 0.97
+2020-04-02 08:26:47,273-INFO: detect dog at [127.44086, 215.71997, 316.04276, 539.7584] score: 0.99
+2020-04-02 08:26:47,274-INFO: detect car at [475.42343, 80.007484, 687.16095, 171.27374] score: 0.98
+2020-04-02 08:26:47,274-INFO: Detection bbox results save in output/dog.jpg
+```
+
+## 参考论文
+
+- [You Only Look Once: Unified, Real-Time Object Detection](https://arxiv.org/abs/1506.02640v5), Joseph Redmon, Santosh Divvala, Ross Girshick, Ali Farhadi.
+- [YOLOv3: An Incremental Improvement](https://arxiv.org/abs/1804.02767v1), Joseph Redmon, Ali Farhadi.
+- [Bag of Freebies for Training Object Detection Neural Networks](https://arxiv.org/abs/1902.04103v3), Zhi Zhang, Tong He, Hang Zhang, Zhongyue Zhang, Junyuan Xie, Mu Li.
+
diff --git a/yolov3/__init__.py b/yolov3/__init__.py
deleted file mode 100644
index 9118340d83fefa17d4a7e8fc577ee22a2d3a2656..0000000000000000000000000000000000000000
--- a/yolov3/__init__.py
+++ /dev/null
@@ -1,14 +0,0 @@
-# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
diff --git a/yolov3/coco.py b/yolov3/coco.py
new file mode 100644
index 0000000000000000000000000000000000000000..34809246c1f90d3ad029842c19ae5f2c3eba08b0
--- /dev/null
+++ b/yolov3/coco.py
@@ -0,0 +1,275 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import division
+from __future__ import print_function
+
+import os
+import cv2
+import numpy as np
+from pycocotools.coco import COCO
+
+from paddle.fluid.io import Dataset
+
+import logging
+logger = logging.getLogger(__name__)
+
+__all__ = ['COCODataset']
+
+
+class COCODataset(Dataset):
+ """
+ Load dataset with MS-COCO format.
+
+ Args:
+ dataset_dir (str): root directory for dataset.
+ image_dir (str): directory for images.
+ anno_path (str): voc annotation file path.
+ sample_num (int): number of samples to load, -1 means all.
+ use_default_label (bool): whether use the default mapping of
+ label to integer index. Default True.
+ with_background (bool): whether load background as a class,
+ default True.
+ transform (callable): callable transform to perform on samples,
+ default None.
+ mixup (bool): whether return image mixup samples, default False.
+ alpha (float): alpha factor of beta distribution to generate
+ mixup score, used only when mixup is True, default 1.5
+ beta (float): beta factor of beta distribution to generate
+ mixup score, used only when mixup is True, default 1.5
+ """
+
+ def __init__(self,
+ dataset_dir='',
+ image_dir='',
+ anno_path='',
+ sample_num=-1,
+ with_background=True,
+ transform=None,
+ mixup=False,
+ alpha=1.5,
+ beta=1.5):
+ # roidbs is list of dict whose structure is:
+ # {
+ # 'im_file': im_fname, # image file name
+ # 'im_id': im_id, # image id
+ # 'h': im_h, # height of image
+ # 'w': im_w, # width
+ # 'is_crowd': is_crowd,
+ # 'gt_class': gt_class,
+ # 'gt_bbox': gt_bbox,
+ # 'gt_score': gt_score,
+ # 'difficult': difficult
+ # }
+
+ self._anno_path = os.path.join(dataset_dir, anno_path)
+ self._image_dir = os.path.join(dataset_dir, image_dir)
+ assert os.path.exists(self._anno_path), \
+ "anno_path {} not exists".format(anno_path)
+ assert os.path.exists(self._image_dir), \
+ "image_dir {} not exists".format(image_dir)
+
+ self._sample_num = sample_num
+ self._with_background = with_background
+ self._transform = transform
+ self._mixup = mixup
+ self._alpha = alpha
+ self._beta = beta
+
+ # load in dataset roidbs
+ self._load_roidb_and_cname2cid()
+
+ def _load_roidb_and_cname2cid(self):
+ assert self._anno_path.endswith('.json'), \
+ 'invalid coco annotation file: ' + anno_path
+ coco = COCO(self._anno_path)
+ img_ids = coco.getImgIds()
+ cat_ids = coco.getCatIds()
+ records = []
+ ct = 0
+
+ # when with_background = True, mapping category to classid, like:
+ # background:0, first_class:1, second_class:2, ...
+ catid2clsid = dict({
+ catid: i + int(self._with_background)
+ for i, catid in enumerate(cat_ids)
+ })
+ cname2cid = dict({
+ coco.loadCats(catid)[0]['name']: clsid
+ for catid, clsid in catid2clsid.items()
+ })
+
+ for img_id in img_ids:
+ img_anno = coco.loadImgs(img_id)[0]
+ im_fname = img_anno['file_name']
+ im_w = float(img_anno['width'])
+ im_h = float(img_anno['height'])
+
+ ins_anno_ids = coco.getAnnIds(imgIds=img_id, iscrowd=False)
+ instances = coco.loadAnns(ins_anno_ids)
+
+ bboxes = []
+ for inst in instances:
+ x, y, box_w, box_h = inst['bbox']
+ x1 = max(0, x)
+ y1 = max(0, y)
+ x2 = min(im_w - 1, x1 + max(0, box_w - 1))
+ y2 = min(im_h - 1, y1 + max(0, box_h - 1))
+ if inst['area'] > 0 and x2 >= x1 and y2 >= y1:
+ inst['clean_bbox'] = [x1, y1, x2, y2]
+ bboxes.append(inst)
+ else:
+ logger.warn(
+ 'Found an invalid bbox in annotations: im_id: {}, '
+ 'area: {} x1: {}, y1: {}, x2: {}, y2: {}.'.format(
+ img_id, float(inst['area']), x1, y1, x2, y2))
+ num_bbox = len(bboxes)
+
+ gt_bbox = np.zeros((num_bbox, 4), dtype=np.float32)
+ gt_class = np.zeros((num_bbox, 1), dtype=np.int32)
+ gt_score = np.ones((num_bbox, 1), dtype=np.float32)
+ is_crowd = np.zeros((num_bbox, 1), dtype=np.int32)
+ difficult = np.zeros((num_bbox, 1), dtype=np.int32)
+ gt_poly = [None] * num_bbox
+
+ for i, box in enumerate(bboxes):
+ catid = box['category_id']
+ gt_class[i][0] = catid2clsid[catid]
+ gt_bbox[i, :] = box['clean_bbox']
+ is_crowd[i][0] = box['iscrowd']
+ if 'segmentation' in box:
+ gt_poly[i] = box['segmentation']
+
+ im_fname = os.path.join(self._image_dir,
+ im_fname) if self._image_dir else im_fname
+ coco_rec = {
+ 'im_file': im_fname,
+ 'im_id': np.array([img_id]),
+ 'h': im_h,
+ 'w': im_w,
+ 'is_crowd': is_crowd,
+ 'gt_class': gt_class,
+ 'gt_bbox': gt_bbox,
+ 'gt_score': gt_score,
+ 'gt_poly': gt_poly,
+ }
+
+ records.append(coco_rec)
+ ct += 1
+ if self._sample_num > 0 and ct >= self._sample_num:
+ break
+ assert len(records) > 0, 'not found any coco record in %s' % (self._anno_path)
+ logger.info('{} samples in file {}'.format(ct, self._anno_path))
+ self._roidbs, self._cname2cid = records, cname2cid
+
+ @property
+ def num_classes(self):
+ return len(self._cname2cid)
+
+ def __len__(self):
+ return len(self._roidbs)
+
+ def _getitem_by_index(self, idx):
+ roidb = self._roidbs[idx]
+ with open(roidb['im_file'], 'rb') as f:
+ data = np.frombuffer(f.read(), dtype='uint8')
+ im = cv2.imdecode(data, 1)
+ im = cv2.cvtColor(im, cv2.COLOR_BGR2RGB)
+ im_info = np.array([roidb['im_id'][0], roidb['h'], roidb['w']], dtype='int32')
+ gt_bbox = roidb['gt_bbox']
+ gt_class = roidb['gt_class']
+ gt_score = roidb['gt_score']
+ return im_info, im, gt_bbox, gt_class, gt_score
+
+ def __getitem__(self, idx):
+ im_info, im, gt_bbox, gt_class, gt_score = self._getitem_by_index(idx)
+
+ if self._mixup:
+ mixup_idx = idx + np.random.randint(1, self.__len__())
+ mixup_idx %= self.__len__()
+ _, mixup_im, mixup_bbox, mixup_class, _ = \
+ self._getitem_by_index(mixup_idx)
+
+ im, gt_bbox, gt_class, gt_score = \
+ self._mixup_image(im, gt_bbox, gt_class, mixup_im,
+ mixup_bbox, mixup_class)
+
+ if self._transform:
+ im_info, im, gt_bbox, gt_class, gt_score = \
+ self._transform(im_info, im, gt_bbox, gt_class, gt_score)
+
+ return [im_info, im, gt_bbox, gt_class, gt_score]
+
+ def _mixup_image(self, img1, bbox1, class1, img2, bbox2, class2):
+ factor = np.random.beta(self._alpha, self._beta)
+ factor = max(0.0, min(1.0, factor))
+ if factor >= 1.0:
+ return img1, bbox1, class1, np.ones_like(class1, dtype="float32")
+ if factor <= 0.0:
+ return img2, bbox2, class2, np.ones_like(class2, dtype="float32")
+
+ h = max(img1.shape[0], img2.shape[0])
+ w = max(img1.shape[1], img2.shape[1])
+ img = np.zeros((h, w, img1.shape[2]), 'float32')
+ img[:img1.shape[0], :img1.shape[1], :] = \
+ img1.astype('float32') * factor
+ img[:img2.shape[0], :img2.shape[1], :] += \
+ img2.astype('float32') * (1.0 - factor)
+
+ gt_bbox = np.concatenate((bbox1, bbox2), axis=0)
+ gt_class = np.concatenate((class1, class2), axis=0)
+
+ score1 = np.ones_like(class1, dtype="float32") * factor
+ score2 = np.ones_like(class2, dtype="float32") * (1.0 - factor)
+ gt_score = np.concatenate((score1, score2), axis=0)
+
+ return img, gt_bbox, gt_class, gt_score
+
+ @property
+ def mixup(self):
+ return self._mixup
+
+ @mixup.setter
+ def mixup(self, value):
+ if not isinstance(value, bool):
+ raise ValueError("mixup should be a boolean number")
+ logger.info("{} set mixup to {}".format(self, value))
+ self._mixup = value
+
+def pascalvoc_label(with_background=True):
+ labels_map = {
+ 'aeroplane': 1,
+ 'bicycle': 2,
+ 'bird': 3,
+ 'boat': 4,
+ 'bottle': 5,
+ 'bus': 6,
+ 'car': 7,
+ 'cat': 8,
+ 'chair': 9,
+ 'cow': 10,
+ 'diningtable': 11,
+ 'dog': 12,
+ 'horse': 13,
+ 'motorbike': 14,
+ 'person': 15,
+ 'pottedplant': 16,
+ 'sheep': 17,
+ 'sofa': 18,
+ 'train': 19,
+ 'tvmonitor': 20
+ }
+ if not with_background:
+ labels_map = {k: v - 1 for k, v in labels_map.items()}
+ return labels_map
diff --git a/yolov3/coco_metric.py b/yolov3/coco_metric.py
index ec7bcac24b3dde91d3ae85e39e7bf9e5151f43ec..2f2f9825b1f90c08afa7b6089641d5a4b28be51d 100644
--- a/yolov3/coco_metric.py
+++ b/yolov3/coco_metric.py
@@ -17,8 +17,6 @@ import json
from pycocotools.cocoeval import COCOeval
from pycocotools.coco import COCO
-from metrics import Metric
-
import logging
FORMAT = '%(asctime)s-%(levelname)s: %(message)s'
logging.basicConfig(level=logging.INFO, format=FORMAT)
@@ -26,12 +24,13 @@ logger = logging.getLogger(__name__)
__all__ = ['COCOMetric']
-
OUTFILE = './bbox.json'
-# considered to change to a callback later
-class COCOMetric(Metric):
+# COCOMetric behavior is different from Metric defined in high
+# level API, COCOMetric will and con only accumulate on the epoch
+# end, so we impliment COCOMetric as not a high level API Metric
+class COCOMetric():
"""
Metrci for MS-COCO dataset, only support update with batch
size as 1.
@@ -43,26 +42,24 @@ class COCOMetric(Metric):
"""
def __init__(self, anno_path, with_background=True, **kwargs):
- super(COCOMetric, self).__init__(**kwargs)
self.anno_path = anno_path
self.with_background = with_background
self.bbox_results = []
self.coco_gt = COCO(anno_path)
cat_ids = self.coco_gt.getCatIds()
- self.clsid2catid = dict(
- {i + int(with_background): catid
- for i, catid in enumerate(cat_ids)})
+ self.clsid2catid = dict(
+ {i + int(with_background): catid
+ for i, catid in enumerate(cat_ids)})
- def update(self, preds, *args, **kwargs):
- im_ids, bboxes = preds
- assert im_ids.shape[0] == 1, \
+ def update(self, img_id, bboxes):
+ assert img_id.shape[0] == 1, \
"COCOMetric can only update with batch size = 1"
if bboxes.shape[1] != 6:
# no bbox detected in this batch
return
- im_id = int(im_ids)
+ img_id = int(img_id)
for i in range(bboxes.shape[0]):
dt = bboxes[i, :]
clsid, score, xmin, ymin, xmax, ymax = dt.tolist()
@@ -72,7 +69,7 @@ class COCOMetric(Metric):
h = ymax - ymin + 1
bbox = [xmin, ymin, w, h]
coco_res = {
- 'image_id': im_id,
+ 'image_id': img_id,
'category_id': catid,
'bbox': bbox,
'score': score
@@ -83,30 +80,30 @@ class COCOMetric(Metric):
self.bbox_results = []
def accumulate(self):
- if len(self.bbox_results) == 0:
- logger.warning("The number of valid bbox detected is zero.\n \
- Please use reasonable model and check input data.\n \
- stop COCOMetric accumulate!")
- return [0.0]
- with open(OUTFILE, 'w') as f:
- json.dump(self.bbox_results, f)
-
- map_stats = self.cocoapi_eval(OUTFILE, 'bbox', coco_gt=self.coco_gt)
- # flush coco evaluation result
- sys.stdout.flush()
+ if len(self.bbox_results) == 0:
+ logger.warning("The number of valid bbox detected is zero.\n \
+ Please use reasonable model and check input data.\n \
+ stop COCOMetric accumulate!")
+ return [0.0]
+ with open(OUTFILE, 'w') as f:
+ json.dump(self.bbox_results, f)
+
+ map_stats = self.cocoapi_eval(OUTFILE, 'bbox', coco_gt=self.coco_gt)
+ # flush coco evaluation result
+ sys.stdout.flush()
self.result = map_stats[0]
- return self.result
+ return [self.result]
def cocoapi_eval(self, jsonfile, style, coco_gt=None, anno_file=None):
- assert coco_gt != None or anno_file != None
-
- if coco_gt == None:
- coco_gt = COCO(anno_file)
- logger.info("Start evaluate...")
- coco_dt = coco_gt.loadRes(jsonfile)
- coco_eval = COCOeval(coco_gt, coco_dt, style)
- coco_eval.evaluate()
- coco_eval.accumulate()
- coco_eval.summarize()
- return coco_eval.stats
+ assert coco_gt != None or anno_file != None
+
+ if coco_gt == None:
+ coco_gt = COCO(anno_file)
+ logger.info("Start evaluate...")
+ coco_dt = coco_gt.loadRes(jsonfile)
+ coco_eval = COCOeval(coco_gt, coco_dt, style)
+ coco_eval.evaluate()
+ coco_eval.accumulate()
+ coco_eval.summarize()
+ return coco_eval.stats
diff --git a/yolov3/dataset/download_voc.py b/yolov3/dataset/download_voc.py
new file mode 100644
index 0000000000000000000000000000000000000000..0d4e3cf368ca446f989f19449adf2775d741fe7f
--- /dev/null
+++ b/yolov3/dataset/download_voc.py
@@ -0,0 +1,46 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+import os.path as osp
+import sys
+import tarfile
+
+from download import _download
+
+import logging
+logger = logging.getLogger(__name__)
+
+DATASETS = {
+ 'voc': [
+ ('https://paddlemodels.bj.bcebos.com/hapi/voc.tar',
+ '9faeb7fd997aeea843092fd608d5bcb4', ),
+ ],
+}
+
+def download_decompress_file(data_dir, url, md5):
+ logger.info("Downloading from {}".format(url))
+ tar_file = _download(url, data_dir, md5)
+ logger.info("Decompressing {}".format(tar_file))
+ with tarfile.open(tar_file) as tf:
+ tf.extractall(path=data_dir)
+ os.remove(tar_file)
+
+
+if __name__ == "__main__":
+ data_dir = osp.split(osp.realpath(sys.argv[0]))[0]
+ for name, infos in DATASETS.items():
+ for info in infos:
+ download_decompress_file(data_dir, *info)
+
diff --git a/yolov3/image/YOLOv3.jpg b/yolov3/image/YOLOv3.jpg
new file mode 100644
index 0000000000000000000000000000000000000000..06b81f545247c1d542fd661f947eb0cf3edc480e
Binary files /dev/null and b/yolov3/image/YOLOv3.jpg differ
diff --git a/yolov3/image/YOLOv3_structure.jpg b/yolov3/image/YOLOv3_structure.jpg
new file mode 100644
index 0000000000000000000000000000000000000000..51bd2d1733e2f78945d3e871cb5b649aad95d633
Binary files /dev/null and b/yolov3/image/YOLOv3_structure.jpg differ
diff --git a/yolov3/image/dog.jpg b/yolov3/image/dog.jpg
new file mode 100644
index 0000000000000000000000000000000000000000..77b0381222eaed50867643f4166092c781e56d5b
Binary files /dev/null and b/yolov3/image/dog.jpg differ
diff --git a/yolov3/infer.py b/yolov3/infer.py
new file mode 100644
index 0000000000000000000000000000000000000000..f19e86615a0b1c8c57f3469f5a5bdcaa85535e9c
--- /dev/null
+++ b/yolov3/infer.py
@@ -0,0 +1,126 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import division
+from __future__ import print_function
+
+import os
+import argparse
+import numpy as np
+from PIL import Image
+
+from paddle import fluid
+from paddle.fluid.optimizer import Momentum
+from paddle.fluid.io import DataLoader
+
+from model import Model, Input, set_device
+from models import yolov3_darknet53, YoloLoss
+
+from coco import COCODataset
+from transforms import *
+from visualizer import draw_bbox
+
+import logging
+logger = logging.getLogger(__name__)
+
+IMAGE_MEAN = [0.485, 0.456, 0.406]
+IMAGE_STD = [0.229, 0.224, 0.225]
+
+
+def get_save_image_name(output_dir, image_path):
+ """
+ Get save image name from source image path.
+ """
+ if not os.path.exists(output_dir):
+ os.makedirs(output_dir)
+ image_name = os.path.split(image_path)[-1]
+ name, ext = os.path.splitext(image_name)
+ return os.path.join(output_dir, "{}".format(name)) + ext
+
+
+def load_labels(label_list, with_background=True):
+ idx = int(with_background)
+ cat2name = {}
+ with open(label_list) as f:
+ for line in f.readlines():
+ line = line.strip()
+ if line:
+ cat2name[idx] = line
+ idx += 1
+ return cat2name
+
+
+def main():
+ device = set_device(FLAGS.device)
+ fluid.enable_dygraph(device) if FLAGS.dynamic else None
+
+ inputs = [Input([None, 3], 'int32', name='img_info'),
+ Input([None, 3, None, None], 'float32', name='image')]
+
+ cat2name = load_labels(FLAGS.label_list, with_background=False)
+
+ model = yolov3_darknet53(num_classes=len(cat2name),
+ model_mode='test',
+ pretrained=FLAGS.weights is None)
+
+ model.prepare(inputs=inputs, device=FLAGS.device)
+
+ if FLAGS.weights is not None:
+ model.load(FLAGS.weights, reset_optimizer=True)
+
+ # image preprocess
+ orig_img = Image.open(FLAGS.infer_image).convert('RGB')
+ w, h = orig_img.size
+ img = orig_img.resize((608, 608), Image.BICUBIC)
+ img = np.array(img).astype('float32') / 255.0
+ img -= np.array(IMAGE_MEAN)
+ img /= np.array(IMAGE_STD)
+ img = img.transpose((2, 0, 1))[np.newaxis, :]
+ img_info = np.array([0, h, w]).astype('int32')[np.newaxis, :]
+
+ _, bboxes = model.test([img_info, img])
+
+ vis_img = draw_bbox(orig_img, cat2name, bboxes, FLAGS.draw_threshold)
+ save_name = get_save_image_name(FLAGS.output_dir, FLAGS.infer_image)
+ logger.info("Detection bbox results save in {}".format(save_name))
+ vis_img.save(save_name, quality=95)
+
+
+if __name__ == '__main__':
+ parser = argparse.ArgumentParser("Yolov3 Training on VOC")
+ parser.add_argument(
+ "--device", type=str, default='gpu', help="device to use, gpu or cpu")
+ parser.add_argument(
+ "-d", "--dynamic", action='store_true', help="enable dygraph mode")
+ parser.add_argument(
+ "--label_list", type=str, default=None,
+ help="path to category label list file")
+ parser.add_argument(
+ "-t", "--draw_threshold", type=float, default=0.5,
+ help="threshold to reserve the result for visualization")
+ parser.add_argument(
+ "-i", "--infer_image", type=str, default=None,
+ help="image path for inference")
+ parser.add_argument(
+ "-o", "--output_dir", type=str, default='output',
+ help="directory to save inference result if --visualize is set")
+ parser.add_argument(
+ "-w", "--weights", default=None, type=str,
+ help="path to weights for inference")
+ FLAGS = parser.parse_args()
+ assert os.path.isfile(FLAGS.infer_image), \
+ "infer_image {} not a file".format(FLAGS.infer_image)
+ assert os.path.isfile(FLAGS.label_list), \
+ "label_list {} not a file".format(FLAGS.label_list)
+ main()
diff --git a/yolov3/main.py b/yolov3/main.py
new file mode 100644
index 0000000000000000000000000000000000000000..18c24d196877586475f6aba1f949c3207665fcce
--- /dev/null
+++ b/yolov3/main.py
@@ -0,0 +1,208 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import division
+from __future__ import print_function
+
+import argparse
+import contextlib
+import os
+
+import numpy as np
+
+from paddle import fluid
+from paddle.fluid.optimizer import Momentum
+from paddle.fluid.io import DataLoader
+
+from model import Model, Input, set_device
+from distributed import DistributedBatchSampler
+from models import yolov3_darknet53, YoloLoss
+
+from coco_metric import COCOMetric
+from coco import COCODataset
+from transforms import *
+
+NUM_MAX_BOXES = 50
+
+
+def make_optimizer(step_per_epoch, parameter_list=None):
+ base_lr = FLAGS.lr
+ warm_up_iter = 1000
+ momentum = 0.9
+ weight_decay = 5e-4
+ boundaries = [step_per_epoch * e for e in [200, 250]]
+ values = [base_lr * (0.1 ** i) for i in range(len(boundaries) + 1)]
+ learning_rate = fluid.layers.piecewise_decay(
+ boundaries=boundaries,
+ values=values)
+ learning_rate = fluid.layers.linear_lr_warmup(
+ learning_rate=learning_rate,
+ warmup_steps=warm_up_iter,
+ start_lr=0.0,
+ end_lr=base_lr)
+ optimizer = fluid.optimizer.Momentum(
+ learning_rate=learning_rate,
+ regularization=fluid.regularizer.L2Decay(weight_decay),
+ momentum=momentum,
+ parameter_list=parameter_list)
+ return optimizer
+
+
+def main():
+ device = set_device(FLAGS.device)
+ fluid.enable_dygraph(device) if FLAGS.dynamic else None
+
+ inputs = [Input([None, 3], 'int32', name='img_info'),
+ Input([None, 3, None, None], 'float32', name='image')]
+ labels = [Input([None, NUM_MAX_BOXES, 4], 'float32', name='gt_bbox'),
+ Input([None, NUM_MAX_BOXES], 'int32', name='gt_label'),
+ Input([None, NUM_MAX_BOXES], 'float32', name='gt_score')]
+
+ if not FLAGS.eval_only: # training mode
+ train_transform = Compose([ColorDistort(),
+ RandomExpand(),
+ RandomCrop(),
+ RandomFlip(),
+ NormalizeBox(),
+ PadBox(),
+ BboxXYXY2XYWH()])
+ train_collate_fn = BatchCompose([RandomShape(), NormalizeImage()])
+ dataset = COCODataset(dataset_dir=FLAGS.data,
+ anno_path='annotations/instances_train2017.json',
+ image_dir='train2017',
+ with_background=False,
+ mixup=True,
+ transform=train_transform)
+ batch_sampler = DistributedBatchSampler(dataset,
+ batch_size=FLAGS.batch_size,
+ shuffle=True,
+ drop_last=True)
+ loader = DataLoader(dataset,
+ batch_sampler=batch_sampler,
+ places=device,
+ num_workers=FLAGS.num_workers,
+ return_list=True,
+ collate_fn=train_collate_fn)
+ else: # evaluation mode
+ eval_transform = Compose([ResizeImage(target_size=608),
+ NormalizeBox(),
+ PadBox(),
+ BboxXYXY2XYWH()])
+ eval_collate_fn = BatchCompose([NormalizeImage()])
+ dataset = COCODataset(dataset_dir=FLAGS.data,
+ anno_path='annotations/instances_val2017.json',
+ image_dir='val2017',
+ with_background=False,
+ transform=eval_transform)
+ # batch_size can only be 1 in evaluation for YOLOv3
+ # prediction bbox is a LoDTensor
+ batch_sampler = DistributedBatchSampler(dataset,
+ batch_size=1,
+ shuffle=False,
+ drop_last=False)
+ loader = DataLoader(dataset,
+ batch_sampler=batch_sampler,
+ places=device,
+ num_workers=FLAGS.num_workers,
+ return_list=True,
+ collate_fn=eval_collate_fn)
+
+ pretrained = FLAGS.eval_only and FLAGS.weights is None
+ model = yolov3_darknet53(num_classes=dataset.num_classes,
+ model_mode='eval' if FLAGS.eval_only else 'train',
+ pretrained=pretrained)
+
+ if FLAGS.pretrain_weights is not None:
+ model.load(FLAGS.pretrain_weights, skip_mismatch=True, reset_optimizer=True)
+
+ optim = make_optimizer(len(batch_sampler), parameter_list=model.parameters())
+
+ model.prepare(optim,
+ YoloLoss(num_classes=dataset.num_classes),
+ inputs=inputs, labels=labels,
+ device=FLAGS.device)
+
+ # NOTE: we implement COCO metric of YOLOv3 model here, separately
+ # from 'prepare' and 'fit' framework for follwing reason:
+ # 1. YOLOv3 network structure is different between 'train' and
+ # 'eval' mode, in 'eval' mode, output prediction bbox is not the
+ # feature map used for YoloLoss calculating
+ # 2. COCO metric behavior is also different from defined Metric
+ # for COCO metric should not perform accumulate in each iteration
+ # but only accumulate at the end of an epoch
+ if FLAGS.eval_only:
+ if FLAGS.weights is not None:
+ model.load(FLAGS.weights, reset_optimizer=True)
+ preds = model.predict(loader, stack_outputs=False)
+ _, _, _, img_ids, bboxes = preds
+
+ anno_path = os.path.join(FLAGS.data, 'annotations/instances_val2017.json')
+ coco_metric = COCOMetric(anno_path=anno_path, with_background=False)
+ for img_id, bbox in zip(img_ids, bboxes):
+ coco_metric.update(img_id, bbox)
+ coco_metric.accumulate()
+ coco_metric.reset()
+ return
+
+ if FLAGS.resume is not None:
+ model.load(FLAGS.resume)
+
+ model.fit(train_data=loader,
+ epochs=FLAGS.epoch - FLAGS.no_mixup_epoch,
+ save_dir="yolo_checkpoint/mixup",
+ save_freq=10)
+
+ # do not use image mixup transfrom in laste FLAGS.no_mixup_epoch epoches
+ dataset.mixup = False
+ model.fit(train_data=loader,
+ epochs=FLAGS.no_mixup_epoch,
+ save_dir="yolo_checkpoint/no_mixup",
+ save_freq=5)
+
+
+if __name__ == '__main__':
+ parser = argparse.ArgumentParser("Yolov3 Training on VOC")
+ parser.add_argument(
+ "--data", type=str, default='dataset/voc',
+ help="path to dataset directory")
+ parser.add_argument(
+ "--device", type=str, default='gpu', help="device to use, gpu or cpu")
+ parser.add_argument(
+ "-d", "--dynamic", action='store_true', help="enable dygraph mode")
+ parser.add_argument(
+ "--eval_only", action='store_true', help="run evaluation only")
+ parser.add_argument(
+ "-e", "--epoch", default=300, type=int, help="number of epoch")
+ parser.add_argument(
+ "--no_mixup_epoch", default=30, type=int,
+ help="number of the last N epoch without image mixup")
+ parser.add_argument(
+ '--lr', '--learning-rate', default=0.001, type=float, metavar='LR',
+ help='initial learning rate')
+ parser.add_argument(
+ "-b", "--batch_size", default=8, type=int, help="batch size")
+ parser.add_argument(
+ "-j", "--num_workers", default=4, type=int, help="reader worker number")
+ parser.add_argument(
+ "-p", "--pretrain_weights", default=None, type=str,
+ help="path to pretrained weights")
+ parser.add_argument(
+ "-r", "--resume", default=None, type=str,
+ help="path to model weights")
+ parser.add_argument(
+ "-w", "--weights", default=None, type=str,
+ help="path to weights for evaluation")
+ FLAGS = parser.parse_args()
+ assert FLAGS.data, "error: must provide data path"
+ main()
diff --git a/yolov3/transforms.py b/yolov3/transforms.py
new file mode 100644
index 0000000000000000000000000000000000000000..a5fbe46cbbfdb39efe3025a351b407b82dbf33c4
--- /dev/null
+++ b/yolov3/transforms.py
@@ -0,0 +1,620 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import division
+from __future__ import print_function
+
+import cv2
+import traceback
+import numpy as np
+
+import logging
+logger = logging.getLogger(__name__)
+
+__all__ = ['ColorDistort', 'RandomExpand', 'RandomCrop', 'RandomFlip',
+ 'NormalizeBox', 'PadBox', 'RandomShape', 'NormalizeImage',
+ 'BboxXYXY2XYWH', 'ResizeImage', 'Compose', 'BatchCompose']
+
+
+class Compose(object):
+ def __init__(self, transforms=[]):
+ self.transforms = transforms
+
+ def __call__(self, *data):
+ for f in self.transforms:
+ try:
+ data = f(*data)
+ except Exception as e:
+ stack_info = traceback.format_exc()
+ logger.info("fail to perform transform [{}] with error: "
+ "{} and stack:\n{}".format(f, e, str(stack_info)))
+ raise e
+ return data
+
+
+class BatchCompose(object):
+ def __init__(self, transforms=[]):
+ self.transforms = transforms
+
+ def __call__(self, data):
+ for f in self.transforms:
+ try:
+ data = f(data)
+ except Exception as e:
+ stack_info = traceback.format_exc()
+ logger.info("fail to perform batch transform [{}] with error: "
+ "{} and stack:\n{}".format(f, e, str(stack_info)))
+ raise e
+
+ # sample list to batch data
+ batch = list(zip(*data))
+
+ return batch
+
+
+class ColorDistort(object):
+ """Random color distortion.
+
+ Args:
+ hue (list): hue settings.
+ in [lower, upper, probability] format.
+ saturation (list): saturation settings.
+ in [lower, upper, probability] format.
+ contrast (list): contrast settings.
+ in [lower, upper, probability] format.
+ brightness (list): brightness settings.
+ in [lower, upper, probability] format.
+ random_apply (bool): whether to apply in random (yolo) or fixed (SSD)
+ order.
+ """
+
+ def __init__(self,
+ hue=[-18, 18, 0.5],
+ saturation=[0.5, 1.5, 0.5],
+ contrast=[0.5, 1.5, 0.5],
+ brightness=[0.5, 1.5, 0.5],
+ random_apply=True):
+ self.hue = hue
+ self.saturation = saturation
+ self.contrast = contrast
+ self.brightness = brightness
+ self.random_apply = random_apply
+
+ def apply_hue(self, img):
+ low, high, prob = self.hue
+ if np.random.uniform(0., 1.) < prob:
+ return img
+
+ img = img.astype(np.float32)
+
+ # XXX works, but result differ from HSV version
+ delta = np.random.uniform(low, high)
+ u = np.cos(delta * np.pi)
+ w = np.sin(delta * np.pi)
+ bt = np.array([[1.0, 0.0, 0.0], [0.0, u, -w], [0.0, w, u]])
+ tyiq = np.array([[0.299, 0.587, 0.114], [0.596, -0.274, -0.321],
+ [0.211, -0.523, 0.311]])
+ ityiq = np.array([[1.0, 0.956, 0.621], [1.0, -0.272, -0.647],
+ [1.0, -1.107, 1.705]])
+ t = np.dot(np.dot(ityiq, bt), tyiq).T
+ img = np.dot(img, t)
+ return img
+
+ def apply_saturation(self, img):
+ low, high, prob = self.saturation
+ if np.random.uniform(0., 1.) < prob:
+ return img
+ delta = np.random.uniform(low, high)
+
+ img = img.astype(np.float32)
+ gray = img * np.array([[[0.299, 0.587, 0.114]]], dtype=np.float32)
+ gray = gray.sum(axis=2, keepdims=True)
+ gray *= (1.0 - delta)
+ img *= delta
+ img += gray
+ return img
+
+ def apply_contrast(self, img):
+ low, high, prob = self.contrast
+ if np.random.uniform(0., 1.) < prob:
+ return img
+ delta = np.random.uniform(low, high)
+
+ img = img.astype(np.float32)
+ img *= delta
+ return img
+
+ def apply_brightness(self, img):
+ low, high, prob = self.brightness
+ if np.random.uniform(0., 1.) < prob:
+ return img
+ delta = np.random.uniform(low, high)
+
+ img = img.astype(np.float32)
+ img += delta
+ return img
+
+ def __call__(self, im_info, im, gt_bbox, gt_class, gt_score):
+ if self.random_apply:
+ distortions = np.random.permutation([
+ self.apply_brightness, self.apply_contrast,
+ self.apply_saturation, self.apply_hue
+ ])
+ for func in distortions:
+ im = func(im)
+ return [im_info, im, gt_bbox, gt_class, gt_score]
+
+ im = self.apply_brightness(im)
+
+ if np.random.randint(0, 2):
+ im = self.apply_contrast(im)
+ im = self.apply_saturation(im)
+ im = self.apply_hue(im)
+ else:
+ im = self.apply_saturation(im)
+ im = self.apply_hue(im)
+ im = self.apply_contrast(im)
+ return [im_info, im, gt_bbox, gt_class, gt_score]
+
+
+class RandomExpand(object):
+ """Random expand the canvas.
+
+ Args:
+ ratio (float): maximum expansion ratio.
+ prob (float): probability to expand.
+ fill_value (list): color value used to fill the canvas. in RGB order.
+ """
+
+ def __init__(self, ratio=4., prob=0.5, fill_value=[123.675, 116.28, 103.53]):
+ assert ratio > 1.01, "expand ratio must be larger than 1.01"
+ self.ratio = ratio
+ self.prob = prob
+ self.fill_value = fill_value
+
+ def __call__(self, im_info, im, gt_bbox, gt_class, gt_score):
+ if np.random.uniform(0., 1.) < self.prob:
+ return [im_info, im, gt_bbox, gt_class, gt_score]
+
+ height, width, _ = im.shape
+ expand_ratio = np.random.uniform(1., self.ratio)
+ h = int(height * expand_ratio)
+ w = int(width * expand_ratio)
+ if not h > height or not w > width:
+ return [im_info, im, gt_bbox, gt_class, gt_score]
+ y = np.random.randint(0, h - height)
+ x = np.random.randint(0, w - width)
+ canvas = np.ones((h, w, 3), dtype=np.uint8)
+ canvas *= np.array(self.fill_value, dtype=np.uint8)
+ canvas[y:y + height, x:x + width, :] = im.astype(np.uint8)
+
+ gt_bbox += np.array([x, y, x, y], dtype=np.float32)
+
+ return [im_info, canvas, gt_bbox, gt_class, gt_score]
+
+
+class RandomCrop():
+ """Random crop image and bboxes.
+
+ Args:
+ aspect_ratio (list): aspect ratio of cropped region.
+ in [min, max] format.
+ thresholds (list): iou thresholds for decide a valid bbox crop.
+ scaling (list): ratio between a cropped region and the original image.
+ in [min, max] format.
+ num_attempts (int): number of tries before giving up.
+ allow_no_crop (bool): allow return without actually cropping them.
+ cover_all_box (bool): ensure all bboxes are covered in the final crop.
+ """
+
+ def __init__(self,
+ aspect_ratio=[.5, 2.],
+ thresholds=[.0, .1, .3, .5, .7, .9],
+ scaling=[.3, 1.],
+ num_attempts=50,
+ allow_no_crop=True,
+ cover_all_box=False):
+ self.aspect_ratio = aspect_ratio
+ self.thresholds = thresholds
+ self.scaling = scaling
+ self.num_attempts = num_attempts
+ self.allow_no_crop = allow_no_crop
+ self.cover_all_box = cover_all_box
+
+ def __call__(self, im_info, im, gt_bbox, gt_class, gt_score):
+ if len(gt_bbox) == 0:
+ return [im_info, im, gt_bbox, gt_class, gt_score]
+
+ # NOTE Original method attempts to generate one candidate for each
+ # threshold then randomly sample one from the resulting list.
+ # Here a short circuit approach is taken, i.e., randomly choose a
+ # threshold and attempt to find a valid crop, and simply return the
+ # first one found.
+ # The probability is not exactly the same, kinda resembling the
+ # "Monty Hall" problem. Actually carrying out the attempts will affect
+ # observability (just like opening doors in the "Monty Hall" game).
+ thresholds = list(self.thresholds)
+ if self.allow_no_crop:
+ thresholds.append('no_crop')
+ np.random.shuffle(thresholds)
+
+ for thresh in thresholds:
+ if thresh == 'no_crop':
+ return [im_info, im, gt_bbox, gt_class, gt_score]
+
+ h, w, _ = im.shape
+ found = False
+ for i in range(self.num_attempts):
+ scale = np.random.uniform(*self.scaling)
+ min_ar, max_ar = self.aspect_ratio
+ aspect_ratio = np.random.uniform(
+ max(min_ar, scale**2), min(max_ar, scale**-2))
+ crop_h = int(h * scale / np.sqrt(aspect_ratio))
+ crop_w = int(w * scale * np.sqrt(aspect_ratio))
+ crop_y = np.random.randint(0, h - crop_h)
+ crop_x = np.random.randint(0, w - crop_w)
+ crop_box = [crop_x, crop_y, crop_x + crop_w, crop_y + crop_h]
+ iou = self._iou_matrix(
+ gt_bbox, np.array(
+ [crop_box], dtype=np.float32))
+ if iou.max() < thresh:
+ continue
+
+ if self.cover_all_box and iou.min() < thresh:
+ continue
+
+ cropped_box, valid_ids = self._crop_box_with_center_constraint(
+ gt_bbox, np.array(
+ crop_box, dtype=np.float32))
+ if valid_ids.size > 0:
+ found = True
+ break
+
+ if found:
+ im = self._crop_image(im, crop_box)
+ gt_bbox = np.take(cropped_box, valid_ids, axis=0)
+ gt_class = np.take(gt_class, valid_ids, axis=0)
+ gt_score = np.take(gt_score, valid_ids, axis=0)
+ return [im_info, im, gt_bbox, gt_class, gt_score]
+
+ return [im_info, im, gt_bbox, gt_class, gt_score]
+
+ def _iou_matrix(self, a, b):
+ tl_i = np.maximum(a[:, np.newaxis, :2], b[:, :2])
+ br_i = np.minimum(a[:, np.newaxis, 2:], b[:, 2:])
+
+ area_i = np.prod(br_i - tl_i, axis=2) * (tl_i < br_i).all(axis=2)
+ area_a = np.prod(a[:, 2:] - a[:, :2], axis=1)
+ area_b = np.prod(b[:, 2:] - b[:, :2], axis=1)
+ area_o = (area_a[:, np.newaxis] + area_b - area_i)
+ return area_i / (area_o + 1e-10)
+
+ def _crop_box_with_center_constraint(self, box, crop):
+ cropped_box = box.copy()
+
+ cropped_box[:, :2] = np.maximum(box[:, :2], crop[:2])
+ cropped_box[:, 2:] = np.minimum(box[:, 2:], crop[2:])
+ cropped_box[:, :2] -= crop[:2]
+ cropped_box[:, 2:] -= crop[:2]
+
+ centers = (box[:, :2] + box[:, 2:]) / 2
+ valid = np.logical_and(crop[:2] <= centers,
+ centers < crop[2:]).all(axis=1)
+ valid = np.logical_and(
+ valid, (cropped_box[:, :2] < cropped_box[:, 2:]).all(axis=1))
+
+ return cropped_box, np.where(valid)[0]
+
+ def _crop_image(self, img, crop):
+ x1, y1, x2, y2 = crop
+ return img[y1:y2, x1:x2, :]
+
+
+class RandomFlip():
+ def __init__(self, prob=0.5, is_normalized=False):
+ """
+ Args:
+ prob (float): the probability of flipping image
+ is_normalized (bool): whether the bbox scale to [0,1]
+ """
+ self.prob = prob
+ self.is_normalized = is_normalized
+ if not (isinstance(self.prob, float) and
+ isinstance(self.is_normalized, bool)):
+ raise TypeError("{}: input type is invalid.".format(self))
+
+ def __call__(self, im_info, im, gt_bbox, gt_class, gt_score):
+ """Filp the image and bounding box.
+ Operators:
+ 1. Flip the image numpy.
+ 2. Transform the bboxes' x coordinates.
+ (Must judge whether the coordinates are normalized!)
+ """
+
+ if not isinstance(im, np.ndarray):
+ raise TypeError("{}: image is not a numpy array.".format(self))
+ if len(im.shape) != 3:
+ raise ImageError("{}: image is not 3-dimensional.".format(self))
+ height, width, _ = im.shape
+ if np.random.uniform(0, 1) < self.prob:
+ im = im[:, ::-1, :]
+ if gt_bbox.shape[0] > 0:
+ oldx1 = gt_bbox[:, 0].copy()
+ oldx2 = gt_bbox[:, 2].copy()
+ if self.is_normalized:
+ gt_bbox[:, 0] = 1 - oldx2
+ gt_bbox[:, 2] = 1 - oldx1
+ else:
+ gt_bbox[:, 0] = width - oldx2 - 1
+ gt_bbox[:, 2] = width - oldx1 - 1
+ if gt_bbox.shape[0] != 0 and (
+ gt_bbox[:, 2] < gt_bbox[:, 0]).all():
+ m = "{}: invalid box, x2 should be greater than x1".format(
+ self)
+ raise ValueError(m)
+ return [im_info, im, gt_bbox, gt_class, gt_score]
+
+
+class NormalizeBox(object):
+ """Transform the bounding box's coornidates to [0,1]."""
+
+ def __call__(self, im_info, im, gt_bbox, gt_class, gt_score):
+ height, width, _ = im.shape
+ for i in range(gt_bbox.shape[0]):
+ gt_bbox[i][0] = gt_bbox[i][0] / width
+ gt_bbox[i][1] = gt_bbox[i][1] / height
+ gt_bbox[i][2] = gt_bbox[i][2] / width
+ gt_bbox[i][3] = gt_bbox[i][3] / height
+ return [im_info, im, gt_bbox, gt_class, gt_score]
+
+
+class PadBox(object):
+ def __init__(self, num_max_boxes=50):
+ """
+ Pad zeros to bboxes if number of bboxes is less than num_max_boxes.
+ Args:
+ num_max_boxes (int): the max number of bboxes
+ """
+ self.num_max_boxes = num_max_boxes
+
+ def __call__(self, im_info, im, gt_bbox, gt_class, gt_score):
+ gt_num = min(self.num_max_boxes, len(gt_bbox))
+ num_max = self.num_max_boxes
+
+ pad_bbox = np.zeros((num_max, 4), dtype=np.float32)
+ if gt_num > 0:
+ pad_bbox[:gt_num, :] = gt_bbox[:gt_num, :]
+ gt_bbox = pad_bbox
+
+ pad_class = np.zeros((num_max), dtype=np.int32)
+ if gt_num > 0:
+ pad_class[:gt_num] = gt_class[:gt_num, 0]
+ gt_class = pad_class
+
+ pad_score = np.zeros((num_max), dtype=np.float32)
+ if gt_num > 0:
+ pad_score[:gt_num] = gt_score[:gt_num, 0]
+ gt_score = pad_score
+ return [im_info, im, gt_bbox, gt_class, gt_score]
+
+
+class BboxXYXY2XYWH(object):
+ """
+ Convert bbox XYXY format to XYWH format.
+ """
+
+ def __call__(self, im_info, im, gt_bbox, gt_class, gt_score):
+ gt_bbox[:, 2:4] = gt_bbox[:, 2:4] - gt_bbox[:, :2]
+ gt_bbox[:, :2] = gt_bbox[:, :2] + gt_bbox[:, 2:4] / 2.
+ return [im_info, im, gt_bbox, gt_class, gt_score]
+
+
+class RandomShape(object):
+ """
+ Randomly reshape a batch. If random_inter is True, also randomly
+ select one an interpolation algorithm [cv2.INTER_NEAREST, cv2.INTER_LINEAR,
+ cv2.INTER_AREA, cv2.INTER_CUBIC, cv2.INTER_LANCZOS4]. If random_inter is
+ False, use cv2.INTER_NEAREST.
+
+ Args:
+ sizes (list): list of int, random choose a size from these
+ random_inter (bool): whether to randomly interpolation, defalut true.
+ """
+
+ def __init__(self,
+ sizes=[320, 352, 384, 416, 448, 480, 512, 544, 576, 608],
+ random_inter=True):
+ self.sizes = sizes
+ self.random_inter = random_inter
+ self.interps = [
+ cv2.INTER_NEAREST,
+ cv2.INTER_LINEAR,
+ cv2.INTER_AREA,
+ cv2.INTER_CUBIC,
+ cv2.INTER_LANCZOS4,
+ ] if random_inter else []
+
+ def __call__(self, samples):
+ shape = np.random.choice(self.sizes)
+ method = np.random.choice(self.interps) if self.random_inter \
+ else cv2.INTER_NEAREST
+ for i in range(len(samples)):
+ im = samples[i][1]
+ h, w = im.shape[:2]
+ scale_x = float(shape) / w
+ scale_y = float(shape) / h
+ im = cv2.resize(
+ im, None, None, fx=scale_x, fy=scale_y, interpolation=method)
+ samples[i][1] = im
+ return samples
+
+
+class NormalizeImage(object):
+ def __init__(self,
+ mean=[0.485, 0.456, 0.406],
+ std=[0.229, 0.224, 0.225],
+ scale=True,
+ channel_first=True):
+ """
+ Args:
+ mean (list): the pixel mean
+ std (list): the pixel variance
+ scale (bool): whether scale image to [0, 1]
+ channel_first (bool): whehter change [h, w, c] to [c, h, w]
+ """
+ self.mean = mean
+ self.std = std
+ self.scale = scale
+ self.channel_first = channel_first
+ if not (isinstance(self.mean, list) and isinstance(self.std, list) and
+ isinstance(self.scale, bool)):
+ raise TypeError("{}: input type is invalid.".format(self))
+ from functools import reduce
+ if reduce(lambda x, y: x * y, self.std) == 0:
+ raise ValueError('{}: std is invalid!'.format(self))
+
+ def __call__(self, samples):
+ """Normalize the image.
+ Operators:
+ 1. (optional) Scale the image to [0,1]
+ 2. Each pixel minus mean and is divided by std
+ 3. (optional) permute channel
+ """
+ for i in range(len(samples)):
+ im = samples[i][1]
+ im = im.astype(np.float32, copy=False)
+ mean = np.array(self.mean)[np.newaxis, np.newaxis, :]
+ std = np.array(self.std)[np.newaxis, np.newaxis, :]
+ if self.scale:
+ im = im / 255.0
+ im -= mean
+ im /= std
+ if self.channel_first:
+ im = im.transpose((2, 0, 1))
+ samples[i][1] = im
+ return samples
+
+
+def _iou_matrix(a, b):
+ tl_i = np.maximum(a[:, np.newaxis, :2], b[:, :2])
+ br_i = np.minimum(a[:, np.newaxis, 2:], b[:, 2:])
+ area_i = np.prod(br_i - tl_i, axis=2) * (tl_i < br_i).all(axis=2)
+ area_a = np.prod(a[:, 2:] - a[:, :2], axis=1)
+ area_b = np.prod(b[:, 2:] - b[:, :2], axis=1)
+ area_o = (area_a[:, np.newaxis] + area_b - area_i)
+ return area_i / (area_o + 1e-10)
+
+
+def _crop_box_with_center_constraint(box, crop):
+ cropped_box = box.copy()
+ cropped_box[:, :2] = np.maximum(box[:, :2], crop[:2])
+ cropped_box[:, 2:] = np.minimum(box[:, 2:], crop[2:])
+ cropped_box[:, :2] -= crop[:2]
+ cropped_box[:, 2:] -= crop[:2]
+ centers = (box[:, :2] + box[:, 2:]) / 2
+ valid = np.logical_and(
+ crop[:2] <= centers, centers < crop[2:]).all(axis=1)
+ valid = np.logical_and(
+ valid, (cropped_box[:, :2] < cropped_box[:, 2:]).all(axis=1))
+ return cropped_box, np.where(valid)[0]
+
+
+def random_crop(inputs):
+ aspect_ratios = [.5, 2.]
+ thresholds = [.0, .1, .3, .5, .7, .9]
+ scaling = [.3, 1.]
+
+ img, img_ids, gt_box, gt_label = inputs
+ h, w = img.shape[:2]
+
+ if len(gt_box) == 0:
+ return inputs
+
+ np.random.shuffle(thresholds)
+ for thresh in thresholds:
+ found = False
+ for i in range(50):
+ scale = np.random.uniform(*scaling)
+ min_ar, max_ar = aspect_ratios
+ ar = np.random.uniform(max(min_ar, scale**2),
+ min(max_ar, scale**-2))
+ crop_h = int(h * scale / np.sqrt(ar))
+ crop_w = int(w * scale * np.sqrt(ar))
+ crop_y = np.random.randint(0, h - crop_h)
+ crop_x = np.random.randint(0, w - crop_w)
+ crop_box = [crop_x, crop_y, crop_x + crop_w, crop_y + crop_h]
+ iou = _iou_matrix(gt_box, np.array([crop_box], dtype=np.float32))
+ if iou.max() < thresh:
+ continue
+
+ cropped_box, valid_ids = _crop_box_with_center_constraint(
+ gt_box, np.array(crop_box, dtype=np.float32))
+ if valid_ids.size > 0:
+ found = True
+ break
+
+ if found:
+ x1, y1, x2, y2 = crop_box
+ img = img[y1:y2, x1:x2, :]
+ gt_box = np.take(cropped_box, valid_ids, axis=0)
+ gt_label = np.take(gt_label, valid_ids, axis=0)
+ return img, img_ids, gt_box, gt_label
+
+ return inputs
+
+
+class ResizeImage(object):
+ def __init__(self,
+ target_size=0,
+ interp=cv2.INTER_CUBIC):
+ """
+ Rescale image to the specified target size.
+ If target_size is list, selected a scale randomly as the specified
+ target size.
+
+ Args:
+ target_size (int|list): the target size of image's short side,
+ multi-scale training is adopted when type is list.
+ interp (int): the interpolation method
+ """
+ self.interp = int(interp)
+ if not (isinstance(target_size, int) or isinstance(target_size, list)):
+ raise TypeError(
+ "Type of target_size is invalid. Must be Integer or List, now is {}".
+ format(type(target_size)))
+ self.target_size = target_size
+
+ def __call__(self, im_info, im, gt_bbox, gt_class, gt_score):
+ """ Resize the image numpy.
+ """
+ if not isinstance(im, np.ndarray):
+ raise TypeError("{}: image type is not numpy.".format(self))
+ if len(im.shape) != 3:
+ raise ImageError('{}: image is not 3-dimensional.'.format(self))
+ im_shape = im.shape
+ im_scale_x = float(self.target_size) / float(im_shape[1])
+ im_scale_y = float(self.target_size) / float(im_shape[0])
+ resize_w = self.target_size
+ resize_h = self.target_size
+
+ im = cv2.resize(
+ im,
+ None,
+ None,
+ fx=im_scale_x,
+ fy=im_scale_y,
+ interpolation=self.interp)
+
+ return [im_info, im, gt_bbox, gt_class, gt_score]
+
diff --git a/yolov3/visualizer.py b/yolov3/visualizer.py
new file mode 100644
index 0000000000000000000000000000000000000000..4433df8606ec140fe08f197e445aea6df89bf445
--- /dev/null
+++ b/yolov3/visualizer.py
@@ -0,0 +1,82 @@
+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import division
+from __future__ import print_function
+
+import numpy as np
+from PIL import Image, ImageDraw
+
+import logging
+logger = logging.getLogger(__name__)
+
+__all__ = ['draw_bbox']
+
+
+def color_map(num_classes):
+ color_map = num_classes * [0, 0, 0]
+ for i in range(0, num_classes):
+ j = 0
+ lab = i
+ while lab:
+ color_map[i * 3] |= (((lab >> 0) & 1) << (7 - j))
+ color_map[i * 3 + 1] |= (((lab >> 1) & 1) << (7 - j))
+ color_map[i * 3 + 2] |= (((lab >> 2) & 1) << (7 - j))
+ j += 1
+ lab >>= 3
+ color_map = np.array(color_map).reshape(-1, 3)
+ return color_map
+
+
+def draw_bbox(image, catid2name, bboxes, threshold):
+ """
+ Draw bbox on image
+ """
+ bboxes = np.array(bboxes)
+ if bboxes.shape[1] != 6:
+ logger.info("No bbox detect")
+ return image
+
+ draw = ImageDraw.Draw(image)
+
+ catid2color = {}
+ color_list = color_map(len(catid2name))
+ for bbox in bboxes:
+ catid, score, xmin, ymin, xmax, ymax = bbox
+
+ if score < threshold:
+ continue
+
+ if catid not in catid2color:
+ idx = np.random.randint(len(color_list))
+ catid2color[catid] = color_list[idx]
+ color = tuple(catid2color[catid])
+
+ # draw bbox
+ draw.line(
+ [(xmin, ymin), (xmin, ymax), (xmax, ymax), (xmax, ymin),
+ (xmin, ymin)],
+ width=2,
+ fill=color)
+ logger.info("detect {} at {} score: {:.2f}".format(
+ catid2name[int(catid)], [xmin, ymin, xmax, ymax], score))
+
+ # draw label
+ text = "{} {:.2f}".format(catid2name[catid], score)
+ tw, th = draw.textsize(text)
+ draw.rectangle(
+ [(xmin + 1, ymin - th), (xmin + tw + 1, ymin)], fill=color)
+ draw.text((xmin + 1, ymin - th), text, fill=(255, 255, 255))
+
+ return image
