Skip to content

P
PaddleHub

PaddlePaddle / PaddleHub
1 年多 前同步成功

通知 283
Star 12117
Fork 2091
P
PaddleHub
提交 2ae797eb 编写于 作者: Z Zeyu Chen
浏览文件
操作

merge lastest code

上级 2773c85b 1039a9ac
展开全部 隐藏空白更改
内联 并排
Showing with 546 addition and 22442 deletion
demo/image-classification/README.md
浏览文件 @ 2ae797eb
# PaddleHub图像分类迁移
# PaddleHub 图像分类
## 关于图像分类
https://github.com/PaddlePaddle/models/tree/develop/fluid/PaddleCV/image_classification
## 关于
## 创建Module
本目录包含了创建一个基于ImageNet 2012数据集预训练的图像分类模型(ResNet/MobileNet)的Module的脚本。
通过以下脚本来一键创建一个ResNet50 Module
本示例将展示如何使用PaddleHub Finetune API以及[图像分类](https://github.com/PaddlePaddle/models/tree/develop/fluid/PaddleCV/image_classification)预训练模型完成分类任务。
## 准备工作
在运行本目录的脚本前，需要先安装1.3.0版本以上的PaddlePaddle（如果您本地已经安装了符合条件的PaddlePaddle版本，那么可以跳过`准备工作`这一步）。
如果您的机器支持GPU，我们建议下载GPU版本的PaddlePaddle，使用GPU进行训练和预测的效率都比使用CPU要高。
```shell
sh create_module.sh
# 安装GPU版本的PaddlePaddle
$ pip install --upgrade paddlepaddle-gpu
```
NOTE:
* 如果进行下面示例的脚本或者代码，请确保执行上述脚本
* 关于创建Module的API和细节，请查看`create_module.py`
## 使用Module预测
该Module创建完成后，可以通过命令行或者python API两种方式进行预测
### 命令行方式
`infer.sh`给出了使用命令行调用Module预测的示例脚本
通过以下命令试验下效果
如果您的机器不支持GPU，可以通过下面的命令来安装CPU版本的PaddlePaddle
```shell
sh infer.sh
# 安装CPU版本的PaddlePaddle
$ pip install --upgrade paddlepaddle
```
### 通过python API
`infer_by_code.py`给出了使用python API调用Module预测的示例代码
通过以下命令试验下效果
在安装过程中如果遇到问题，您可以到[Paddle官方网站](http://www.paddlepaddle.org/)上查看解决方案。
## 开始Finetune
在完成安装PaddlePaddle与PaddleHub后，通过执行脚本`sh run_classifier.sh`即可开始使用进行finetune。
脚本支持的参数如下：
```shell
python infer_by_code.py
--batch_size: 批处理大小，请结合显存情况进行调整，若出现显存不足，请适当调低这一参数。默认为16
--num_epoch: finetune迭代的轮数。默认为1
--module: 使用哪个Module作为finetune的特征提取器，脚本支持{resnet50/resnet101/resnet152/mobilenet/nasnet/pnasnet}等模型。默认为resnet50
--checkpoint_dir: 模型保存路径，PaddleHub会自动保存验证集上表现最好的模型。默认为paddlehub_finetune_ckpt
--dataset: 使用什么数据集进行finetune, 脚本支持分别是{flowers/dogcat}。默认为flowers
--use_gpu: 使用使用GPU进行训练，如果机器支持GPU且安装了GPU版本的PaddlePaddle，我们建议您打开这个开关。默认关闭
```
## 对预训练模型进行Finetune
通过以下命令进行Finetune
## 进行预测
当完成finetune后，finetune过程在验证集上表现最优的模型会被保存在`${CHECKPOINT_DIR}/best_model`目录下，其中`${CHECKPOINT_DIR}`目录为finetune时所选择的保存checkpoint的目录。
我们使用该模型来进行预测。执行脚本`sh predict.sh`即可开始使用进行预测。
脚本支持的参数如下：
```shell
sh finetune.sh
--module: 使用哪个Module作为finetune的特征提取器，脚本支持{resnet50/resnet101/resnet152/mobilenet/nasnet/pnasnet}等模型。默认为resnet50
--checkpoint_dir: 模型保存路径，PaddleHub会自动保存验证集上表现最好的模型。默认为paddlehub_finetune_ckpt
--dataset: 使用什么数据集进行finetune, 脚本支持分别是{flowers/dogcat}。默认为flowers
--use_gpu: 使用使用GPU进行训练，如果本机支持GPU且安装了GPU版本的PaddlePaddle，我们建议您打开这个开关。默认关闭
```
更多关于Finetune的资料，请查看[基于PaddleHub的迁移学习](https://github.com/PaddlePaddle/PaddleHub/blob/develop/docs/transfer_learning_turtorial.md)
`注意`：进行预测时，所选择的module，checkpoint_dir，dataset必须和finetune所用的一样
demo/image-classification/img_classifier.py
浏览文件 @ 2ae797eb
......@@ -7,12 +7,12 @@ import numpy as np
# yapf: disable
parser = argparse.ArgumentParser(__doc__)
parser.add_argument("--target", type=str, default="finetune", help="Number of epoches for fine-tuning.")
parser.add_argument("--num_epoch", type=int, default=3, help="Number of epoches for fine-tuning.")
parser.add_argument("--use_gpu", type=bool, default=False, help="Whether use GPU for finetuning or predict")
parser.add_argument("--checkpoint_dir", type=str, default="paddlehub_finetune_ckpt", help="Path to training data.")
parser.add_argument("--num_epoch", type=int, default=1, help="Number of epoches for fine-tuning.")
parser.add_argument("--use_gpu", type=bool, default=False, help="Whether use GPU for fine-tuning.")
parser.add_argument("--checkpoint_dir", type=str, default="paddlehub_finetune_ckpt", help="Path to save log data.")
parser.add_argument("--batch_size", type=int, default=16, help="Total examples' number in batch for training.")
parser.add_argument("--module", type=str, default="resnet50", help="Total examples' number in batch for training.")
parser.add_argument("--module", type=str, default="resnet50", help="Module used as feature extractor.")
parser.add_argument("--dataset", type=str, default="flowers", help="Dataset to finetune.")
# yapf: enable.
module_map = {
......@@ -25,33 +25,25 @@ module_map = {
}
def get_reader(module, dataset=None):
return hub.reader.ImageClassificationReader(
def finetune(args):
module = hub.Module(name=args.module)
input_dict, output_dict, program = module.context(trainable=True)
dataset = hub.dataset.Flowers()
data_reader = hub.reader.ImageClassificationReader(
image_width=module.get_expected_image_width(),
image_height=module.get_expected_image_height(),
images_mean=module.get_pretrained_images_mean(),
images_std=module.get_pretrained_images_std(),
dataset=dataset)
feature_map = output_dict["feature_map"]
task = hub.create_img_cls_task(
feature=feature_map, num_classes=dataset.num_labels)
def get_task(module, num_classes):
input_dict, output_dict, program = module.context(trainable=True)
with fluid.program_guard(program):
img = input_dict["image"]
feature_map = output_dict["feature_map"]
task = hub.create_img_cls_task(
feature=feature_map, num_classes=num_classes)
return task
def finetune(args):
module = hub.Module(name=args.module)
input_dict, output_dict, program = module.context(trainable=True)
dataset = hub.dataset.Flowers()
data_reader = get_reader(module, dataset)
task = get_task(module, dataset.num_labels)
img = input_dict["image"]
feed_list = [img.name, task.variable('label').name]
config = hub.RunConfig(
use_cuda=args.use_gpu,
num_epoch=args.num_epoch,
......@@ -64,54 +56,6 @@ def finetune(args):
task, feed_list=feed_list, data_reader=data_reader, config=config)
def predict(args):
module = hub.Module(name=args.module)
input_dict, output_dict, program = module.context(trainable=True)
data_reader = get_reader(module)
task = get_task(module, 5)
img = input_dict["image"]
feed_list = [img.name]
label_map = {
0: "roses",
1: "tulips",
2: "daisy",
3: "sunflowers",
4: "dandelion"
}
with fluid.program_guard(task.inference_program()):
place = fluid.CUDAPlace(0) if args.use_gpu else fluid.CPUPlace()
exe = fluid.Executor(place)
pretrained_model_dir = os.path.join(args.checkpoint_dir, "best_model")
if not os.path.exists(pretrained_model_dir):
hub.logger.error(
"pretrained model dir %s didn't exist" % pretrained_model_dir)
exit(1)
fluid.io.load_persistables(exe, pretrained_model_dir)
feeder = fluid.DataFeeder(feed_list=feed_list, place=place)
data = ["test/test_img_roses.jpg", "test/test_img_daisy.jpg"]
predict_reader = data_reader.data_generator(
phase="predict", batch_size=1, data=data)
for index, batch in enumerate(predict_reader()):
result, = exe.run(
feed=feeder.feed(batch), fetch_list=[task.variable('probs')])
predict_result = label_map[np.argsort(result[0])[::-1][0]]
print("input %i is %s, and the predict result is %s" %
(index, data[index], predict_result))
def main(args):
if args.target == "finetune":
finetune(args)
elif args.target == "predict":
predict(args)
else:
hub.logger.error("target should in %s" % ["finetune", "predict"])
exit(1)
if __name__ == "__main__":
args = parser.parse_args()
if not args.module in module_map:
......@@ -119,4 +63,4 @@ if __name__ == "__main__":
exit(1)
args.module = module_map[args.module]
main(args)
finetune(args)
demo/image-classification/predict.py 0 → 100644
浏览文件 @ 2ae797eb
import argparse
import os
import paddle.fluid as fluid
import paddlehub as hub
import numpy as np
# yapf: disable
parser = argparse.ArgumentParser(__doc__)
parser.add_argument("--use_gpu", type=bool, default=False, help="Whether use GPU for predict.")
parser.add_argument("--checkpoint_dir", type=str, default="paddlehub_finetune_ckpt", help="Path to save log data.")
parser.add_argument("--module", type=str, default="resnet50", help="Module used as a feature extractor.")
parser.add_argument("--dataset", type=str, default="flowers", help="Dataset to finetune.")
# yapf: enable.
module_map = {
"resnet50": "resnet_v2_50_imagenet",
"resnet101": "resnet_v2_101_imagenet",
"resnet152": "resnet_v2_152_imagenet",
"mobilenet": "mobilenet_v2_imagenet",
"nasnet": "nasnet_imagenet",
"pnasnet": "pnasnet_imagenet"
}
def predict(args):
if args.dataset == "dogcat":
dataset = hub.dataset.DogCat()
elif args.dataset == "flowers":
dataset = hub.dataset.Flowers()
label_map = dataset.label_dict()
num_labels = len(label_map)
module = hub.Module(name=args.module)
input_dict, output_dict, program = module.context()
data_reader = hub.reader.ImageClassificationReader(
image_width=module.get_expected_image_width(),
image_height=module.get_expected_image_height(),
images_mean=module.get_pretrained_images_mean(),
images_std=module.get_pretrained_images_std(),
dataset=None)
img = input_dict["image"]
feature_map = output_dict["feature_map"]
task = hub.create_img_cls_task(feature=feature_map, num_classes=num_labels)
img = input_dict["image"]
feed_list = [img.name]
with fluid.program_guard(task.inference_program()):
place = fluid.CUDAPlace(0) if args.use_gpu else fluid.CPUPlace()
exe = fluid.Executor(place)
pretrained_model_dir = os.path.join(args.checkpoint_dir, "best_model")
if not os.path.exists(pretrained_model_dir):
hub.logger.error(
"pretrained model dir %s didn't exist" % pretrained_model_dir)
exit(1)
fluid.io.load_persistables(exe, pretrained_model_dir)
feeder = fluid.DataFeeder(feed_list=feed_list, place=place)
data = ["test/test_img_roses.jpg", "test/test_img_daisy.jpg"]
predict_reader = data_reader.data_generator(
phase="predict", batch_size=1, data=data)
for index, batch in enumerate(predict_reader()):
result, = exe.run(
feed=feeder.feed(batch), fetch_list=[task.variable('probs')])
predict_result = label_map[np.argsort(result[0])[::-1][0]]
print("input %i is %s, and the predict result is %s" %
(index, data[index], predict_result))
if __name__ == "__main__":
args = parser.parse_args()
if not args.module in module_map:
hub.logger.error("module should in %s" % module_map.keys())
exit(1)
args.module = module_map[args.module]
predict(args)
demo/image-classification/run_classifier.sh
浏览文件 @ 2ae797eb
......@@ -28,4 +28,4 @@ done
export CUDA_VISIBLE_DEVICES=${cuda_visible_devices}
python -u img_classifier.py --target finetune --use_gpu ${use_gpu} --batch_size ${batch_size} --checkpoint_dir ${checkpoint_dir} --num_epoch ${num_epoch} --module ${module}
python -u img_classifier.py --use_gpu ${use_gpu} --batch_size ${batch_size} --checkpoint_dir ${checkpoint_dir} --num_epoch ${num_epoch} --module ${module}
demo/image-classification/run_predict.sh
浏览文件 @ 2ae797eb
......@@ -22,4 +22,4 @@ done
export CUDA_VISIBLE_DEVICES=${cuda_visible_devices}
python -u img_classifier.py --target predict --use_gpu ${use_gpu} --checkpoint_dir ${checkpoint_dir} --module ${module}
python -u predict.py --use_gpu ${use_gpu} --checkpoint_dir ${checkpoint_dir} --module ${module}
demo/lac/README.md
浏览文件 @ 2ae797eb
## 关于LAC
https://github.com/baidu/lac
# LAC
## 创建Module
本目录包含了创建一个基于LAC预训练模型的Module的脚本。
通过以下脚本来一键创建一个LAC Module
## 关于
本示例展示如何使用LAC Module进行预测。
LAC是中文词法分析模型，可以用于进行中文句子的分词/词性标注/命名实体识别等功能，关于模型的训练细节，请查看[LAC](https://github.com/baidu/lac)
## 准备工作
在运行本目录的脚本前，需要先安装1.3.0版本以上的PaddlePaddle（如果您本地已经安装了符合条件的PaddlePaddle版本，那么可以跳过`准备工作`这一步）。
如果您的机器支持GPU，我们建议下载GPU版本的PaddlePaddle，使用GPU进行训练和预测的效率都比使用CPU要高。
```shell
sh create_module.sh
# 安装GPU版本的PaddlePaddle
$ pip install --upgrade paddlepaddle-gpu
```
NOTE:
* 如果进行下面示例的脚本或者代码，请确保执行上述脚本
* 关于创建Module的API和细节，请查看`create_module.py`
## 使用Module预测
该Module创建完成后，可以通过命令行或者python API两种方式进行预测
### 命令行方式
如果您的机器不支持GPU，可以通过下面的命令来安装CPU版本的PaddlePaddle
```shell
# 安装CPU版本的PaddlePaddle
$ pip install --upgrade paddlepaddle
```
在安装过程中如果遇到问题，您可以到[Paddle官方网站](http://www.paddlepaddle.org/)上查看解决方案。
## 命令行方式预测
`infer.sh`给出了使用命令行调用Module预测的示例脚本
通过以下命令试验下效果
```shell
sh infer.sh
$ sh infer.sh
```
### 通过python API
## 通过python API预测
`infer_by_code.py`给出了使用python API调用Module预测的示例代码
通过以下命令试验下效果
```shell
python infer_by_code.py
```
demo/lac/create_module.py 已删除 100644 → 0
浏览文件 @ 2773c85b
import numpy as np
import paddle
import paddle.fluid as fluid
import reader
import paddlehub as hub
import processor
import os
from network import lex_net
def create_module():
word_dict_path = "resources/word.dic"
label_dict_path = "resources/tag.dic"
word_rep_dict_path = "resources/q2b.dic"
pretrained_model = "resources/model"
word2id_dict = reader.load_reverse_dict(word_dict_path)
label2id_dict = reader.load_reverse_dict(label_dict_path)
word_rep_dict = reader.load_dict(word_rep_dict_path)
word_dict_len = max(map(int, word2id_dict.values())) + 1
label_dict_len = max(map(int, label2id_dict.values())) + 1
avg_cost, crf_decode, word, target = lex_net(word_dict_len, label_dict_len)
place = fluid.CPUPlace()
exe = fluid.Executor(place)
exe.run(fluid.default_startup_program())
# load the lac pretrained model
def if_exist(var):
return os.path.exists(os.path.join(pretrained_model, var.name))
fluid.io.load_vars(exe, pretrained_model, predicate=if_exist)
# assets
assets = [word_dict_path, label_dict_path, word_rep_dict_path]
# create a module and save as lac.hub_module
sign = hub.create_signature(
name="lexical_analysis",
inputs=[word],
outputs=[crf_decode],
for_predict=True)
hub.create_module(
sign_arr=[sign],
module_dir="lac.hub_module",
exe=exe,
module_info="resources/module_info.yml",
processor=processor.Processor,
assets=assets)
if __name__ == "__main__":
create_module()
demo/lac/create_module.sh 已删除 100644 → 0
浏览文件 @ 2773c85b
#!/bin/bash
set -o nounset
set -o errexit
script_path=$(cd `dirname $0`; pwd)
module_path=lac.hub_module
if [ -d $script_path/$module_path ]
then
echo "$module_path already existed!"
exit 0
fi
cd $script_path/resources/
if [ ! -d senta_model ]
then
sh download.sh
fi
cd $script_path/
python create_module.py
echo "Successfully create $module_path"
demo/lac/infer.sh
浏览文件 @ 2ae797eb
python ../../paddlehub/commands/hub.py run lac.hub_module/ --signature lexical_analysis --config resources/test/test.yml --input_file resources/test/test.txt
python ../../paddlehub/commands/hub.py run lac --input_file test/test.txt
demo/lac/infer_by_code.py
浏览文件 @ 2ae797eb
......@@ -4,7 +4,7 @@ import paddlehub as hub
def infer_with_input_text():
# get lac module
lac = hub.Module(module_dir="lac.hub_module")
lac = hub.Module(name="lac")
test_text = ["今天是个好日子", "天气预报说今天要下雨", "下一班地铁马上就要到了"]
......@@ -24,14 +24,14 @@ def infer_with_input_text():
def infer_with_input_file():
# get lac module
lac = hub.Module(module_dir="lac.hub_module")
lac = hub.Module(name="lac")
# get the input keys for signature 'lexical_analysis'
data_format = lac.processor.data_format(sign_name='lexical_analysis')
key = list(data_format.keys())[0]
# parse input file
test_file = os.path.join("resources", "test", "test.txt")
test_file = os.path.join("test", "test.txt")
test_text = hub.io.parser.txt_parser.parse(test_file)
# set input dict
......
demo/lac/network.py 已删除 100755 → 0
浏览文件 @ 2773c85b
import sys
import os
import math
import paddle.fluid as fluid
from paddle.fluid.initializer import NormalInitializer
def lex_net(word_dict_len, label_dict_len):
"""
define the lexical analysis network structure
"""
word_emb_dim = 128
grnn_hidden_dim = 256
emb_lr = 5
crf_lr = 0.2
bigru_num = 2
init_bound = 0.1
IS_SPARSE = True
def _bigru_layer(input_feature):
"""
define the bidirectional gru layer
"""
pre_gru = fluid.layers.fc(
input=input_feature,
size=grnn_hidden_dim * 3,
param_attr=fluid.ParamAttr(
initializer=fluid.initializer.Uniform(
low=-init_bound, high=init_bound),
regularizer=fluid.regularizer.L2DecayRegularizer(
regularization_coeff=1e-4)))
gru = fluid.layers.dynamic_gru(
input=pre_gru,
size=grnn_hidden_dim,
param_attr=fluid.ParamAttr(
initializer=fluid.initializer.Uniform(
low=-init_bound, high=init_bound),
regularizer=fluid.regularizer.L2DecayRegularizer(
regularization_coeff=1e-4)))
pre_gru_r = fluid.layers.fc(
input=input_feature,
size=grnn_hidden_dim * 3,
param_attr=fluid.ParamAttr(
initializer=fluid.initializer.Uniform(
low=-init_bound, high=init_bound),
regularizer=fluid.regularizer.L2DecayRegularizer(
regularization_coeff=1e-4)))
gru_r = fluid.layers.dynamic_gru(
input=pre_gru_r,
size=grnn_hidden_dim,
is_reverse=True,
param_attr=fluid.ParamAttr(
initializer=fluid.initializer.Uniform(
low=-init_bound, high=init_bound),
regularizer=fluid.regularizer.L2DecayRegularizer(
regularization_coeff=1e-4)))
bi_merge = fluid.layers.concat(input=[gru, gru_r], axis=1)
return bi_merge
def _net_conf(word, target):
"""
Configure the network
"""
word_embedding = fluid.layers.embedding(
input=word,
size=[word_dict_len, word_emb_dim],
dtype='float32',
is_sparse=IS_SPARSE,
param_attr=fluid.ParamAttr(
learning_rate=emb_lr,
name="word_emb",
initializer=fluid.initializer.Uniform(
low=-init_bound, high=init_bound)))
input_feature = word_embedding
for i in range(bigru_num):
bigru_output = _bigru_layer(input_feature)
input_feature = bigru_output
emission = fluid.layers.fc(
size=label_dict_len,
input=bigru_output,
param_attr=fluid.ParamAttr(
initializer=fluid.initializer.Uniform(
low=-init_bound, high=init_bound),
regularizer=fluid.regularizer.L2DecayRegularizer(
regularization_coeff=1e-4)))
crf_cost = fluid.layers.linear_chain_crf(
input=emission,
label=target,
param_attr=fluid.ParamAttr(name='crfw', learning_rate=crf_lr))
crf_decode = fluid.layers.crf_decoding(
input=emission, param_attr=fluid.ParamAttr(name='crfw'))
avg_cost = fluid.layers.mean(x=crf_cost)
return avg_cost, crf_decode
word = fluid.layers.data(name='word', shape=[1], dtype='int64', lod_level=1)
target = fluid.layers.data(
name="target", shape=[1], dtype='int64', lod_level=1)
avg_cost, crf_decode = _net_conf(word, target)
return avg_cost, crf_decode, word, target
demo/lac/processor.py 已删除 100644 → 0
浏览文件 @ 2773c85b
import paddle
import paddle.fluid as fluid
import paddlehub as hub
import numpy as np
import os
import io
from paddlehub import BaseProcessor
class Processor(BaseProcessor):
def __init__(self, module):
self.module = module
assets_path = self.module.helper.assets_path()
word_dict_path = os.path.join(assets_path, "word.dic")
label_dict_path = os.path.join(assets_path, "tag.dic")
word_rep_dict_path = os.path.join(assets_path, "q2b.dic")
self.id2word_dict = self.load_dict(word_dict_path)
self.word2id_dict = self.load_reverse_dict(word_dict_path)
self.id2label_dict = self.load_dict(label_dict_path)
self.label2id_dict = self.load_reverse_dict(label_dict_path)
self.q2b_dict = self.load_dict(word_rep_dict_path)
def load_dict(self, dict_path):
result_dict = {}
for line in io.open(dict_path, "r", encoding='utf8'):
terms = line.strip("\n").split("\t")
if len(terms) != 2:
continue
result_dict[terms[0]] = terms[1]
return result_dict
def load_reverse_dict(self, dict_path):
result_dict = {}
for line in io.open(dict_path, "r", encoding='utf8'):
terms = line.strip("\n").split("\t")
if len(terms) != 2:
continue
result_dict[terms[1]] = terms[0]
return result_dict
def preprocess(self, sign_name, data_dict):
result = {'text': []}
for sentence in data_dict['text']:
result_i = {}
result_i['origin'] = sentence
line = sentence.strip()
word_idx = []
for word in line:
if ord(word) < 0x20:
word = ' '
if word in self.q2b_dict:
word = self.q2b_dict[word]
if word in self.word2id_dict:
word_idx.append(int(self.word2id_dict[word]))
else:
word_idx.append(int(self.word2id_dict["OOV"]))
result_i['attach'] = line
result_i['processed'] = [x for x in word_idx]
result['text'].append(result_i)
return result
def postprocess(self, sign_name, data_out, data_info, **kwargs):
if sign_name == "lexical_analysis":
result = []
crf_decode = data_out[0]
lod_info = (crf_decode.lod())[0]
np_data = np.array(crf_decode)
for index in range(len(lod_info) - 1):
seg_result = {"word": [], "tag": []}
word_index = 0
outstr = ""
offset = 0
cur_full_word = ""
cur_full_tag = ""
words = data_info['text'][index]['attach']
for tag_index in range(lod_info[index], lod_info[index + 1]):
cur_word = words[word_index]
cur_tag = self.id2label_dict[str(np_data[tag_index][0])]
if cur_tag.endswith("-B") or cur_tag.endswith("O"):
if len(cur_full_word) != 0:
seg_result['word'].append(cur_full_word)
seg_result['tag'].append(cur_full_tag)
cur_full_word = cur_word
cur_full_tag = self.get_real_tag(cur_tag)
else:
cur_full_word += cur_word
word_index += 1
seg_result['word'].append(cur_full_word)
seg_result['tag'].append(cur_full_tag)
result.append(seg_result)
return result
def get_real_tag(self, origin_tag):
if origin_tag == "O":
return "O"
return origin_tag[0:len(origin_tag) - 2]
def data_format(self, sign_name):
if sign_name == "lexical_analysis":
return {
"text": {
"type": hub.DataType.TEXT,
"feed_key": self.module.signatures[sign_name].inputs[0].name
}
}
return None
demo/lac/reader.py 已删除 100644 → 0
浏览文件 @ 2773c85b
"""
The file_reader converts raw corpus to input.
"""
import os
import __future__
import io
def file_reader(file_dir,
word2id_dict,
label2id_dict,
word_replace_dict,
filename_feature=""):
"""
define the reader to read files in file_dir
"""
word_dict_len = max(map(int, word2id_dict.values())) + 1
label_dict_len = max(map(int, label2id_dict.values())) + 1
def reader():
"""
the data generator
"""
index = 0
for root, dirs, files in os.walk(file_dir):
for filename in files:
if not filename.startswith(filename_feature):
continue
for line in io.open(
os.path.join(root, filename), 'r', encoding='utf8'):
index += 1
bad_line = False
line = line.strip("\n")
if len(line) == 0:
continue
seg_tag = line.rfind("\t")
word_part = line[0:seg_tag]
label_part = line[seg_tag + 1:]
word_idx = []
words = word_part
for word in words:
if ord(word) < 0x20:
word = ' '
if word in word_replace_dict:
word = word_replace_dict[word]
if word in word2id_dict:
word_idx.append(int(word2id_dict[word]))
else:
word_idx.append(int(word2id_dict["OOV"]))
target_idx = []
labels = label_part.strip().split(" ")
for label in labels:
if label in label2id_dict:
target_idx.append(int(label2id_dict[label]))
else:
target_idx.append(int(label2id_dict["O"]))
if len(word_idx) != len(target_idx):
continue
yield word_idx, target_idx
return reader
def test_reader(file_dir,
word2id_dict,
label2id_dict,
word_replace_dict,
filename_feature=""):
"""
define the reader to read test files in file_dir
"""
word_dict_len = max(map(int, word2id_dict.values())) + 1
label_dict_len = max(map(int, label2id_dict.values())) + 1
def reader():
"""
the data generator
"""
index = 0
for root, dirs, files in os.walk(file_dir):
for filename in files:
if not filename.startswith(filename_feature):
continue
for line in io.open(
os.path.join(root, filename), 'r', encoding='utf8'):
index += 1
bad_line = False
line = line.strip("\n")
if len(line) == 0:
continue
seg_tag = line.rfind("\t")
if seg_tag == -1:
seg_tag = len(line)
word_part = line[0:seg_tag]
label_part = line[seg_tag + 1:]
word_idx = []
words = word_part
for word in words:
if ord(word) < 0x20:
word = ' '
if word in word_replace_dict:
word = word_replace_dict[word]
if word in word2id_dict:
word_idx.append(int(word2id_dict[word]))
else:
word_idx.append(int(word2id_dict["OOV"]))
yield word_idx, words
return reader
def load_dict(dict_path):
"""
Load a dict. The first column is the key and the second column is the value.
"""
result_dict = {}
for line in io.open(dict_path, "r", encoding='utf8'):
terms = line.strip("\n").split("\t")
if len(terms) != 2:
continue
result_dict[terms[0]] = terms[1]
return result_dict
def load_reverse_dict(dict_path):
"""
Load a dict. The first column is the value and the second column is the key.
"""
result_dict = {}
for line in io.open(dict_path, "r", encoding='utf8'):
terms = line.strip("\n").split("\t")
if len(terms) != 2:
continue
result_dict[terms[1]] = terms[0]
return result_dict
demo/lac/resources/download.sh 已删除 100644 → 0
浏览文件 @ 2773c85b
#!/bin/bash
set -o nounset
set -o errexit
script_path=$(cd `dirname $0`; pwd)
cd $script_path
wget --no-check-certificate https://paddlehub.bj.bcebos.com/paddle_model/lac.tar.gz
tar xvzf lac.tar.gz
rm lac.tar.gz
demo/lac/resources/module_info.yml 已删除 100644 → 0
浏览文件 @ 2773c85b
name: lac
type: nlp/lexical_analysis
author: paddlepaddle
author_email: paddle-dev@baidu.com
summary: "'Lexical Analysis of Chinese', which abbreviated as LAC, is a model used to process lexical analysis. People can use LAC to process Chinese text segmentation, part-of-speech tagging and named entity recognition"
version: 1.0.0
demo/lac/resources/q2b.dic 已删除 100644 → 0
浏览文件 @ 2773c85b
　
、 ,
。 .
— -
～ ~
‖ |
… .
‘ '
’ '
“ "
” "
〔 (
〕 )
〈 <
〉 >
「 '
」 '
『 "
』 "
〖 [
〗 ]
【 [
】 ]
∶ :
＄ $
！ !
＂ "
＃ #
％ %
＆ &
＇ '
（ (
） )
＊ *
＋ +
， ,
－ -
． .
／ /
０ 0
１ 1
２ 2
３ 3
４ 4
５ 5
６ 6
７ 7
８ 8
９ 9
： :
； ;
＜ <
＝ =
＞ >
？ ?
＠ @
Ａ a
Ｂ b
Ｃ c
Ｄ d
Ｅ e
Ｆ f
Ｇ g
Ｈ h
Ｉ i
Ｊ j
Ｋ k
Ｌ l
Ｍ m
Ｎ n
Ｏ o
Ｐ p
Ｑ q
Ｒ r
Ｓ s
Ｔ t
Ｕ u
Ｖ v
Ｗ w
Ｘ x
Ｙ y
Ｚ z
［ [
＼ \
］ ]
＾ ^
＿ _
｀ `
ａ a
ｂ b
ｃ c
ｄ d
ｅ e
ｆ f
ｇ g
ｈ h
ｉ i
ｊ j
ｋ k
ｌ l
ｍ m
ｎ n
ｏ o
ｐ p
ｑ q
ｒ r
ｓ s
ｔ t
ｕ u
ｖ v
ｗ w
ｘ x
ｙ y
ｚ z
｛ {
｜ |
｝ }
￣ ~
〝 "
〞 "
﹐ ,
﹑ ,
﹒ .
﹔ ;
﹕ :
﹖ ?
﹗ !
﹙ (
﹚ )
﹛ {
﹜ {
﹝ [
﹞ ]
﹟ #
﹠ &
﹡ *
﹢ +
﹣ -
﹤ <
﹥ >
﹦ =
﹨ \
﹩ $
﹪ %
﹫ @
,
A a
B b
C c
D d
E e
F f
G g
H h
I i
J j
K k
L l
M m
N n
O o
P p
Q q
R r
S s
T t
U u
V v
W w
X x
Y y
Z z
demo/lac/resources/tag.dic 已删除 100644 → 0
浏览文件 @ 2773c85b
0 a-B
1 a-I
2 ad-B
3 ad-I
4 an-B
5 an-I
6 c-B
7 c-I
8 d-B
9 d-I
10 f-B
11 f-I
12 m-B
13 m-I
14 n-B
15 n-I
16 nr-B
17 nr-I
18 ns-B
19 ns-I
20 nt-B
21 nt-I
22 nw-B
23 nw-I
24 nz-B
25 nz-I
26 p-B
27 p-I
28 q-B
29 q-I
30 r-B
31 r-I
32 s-B
33 s-I
34 t-B
35 t-I
36 u-B
37 u-I
38 v-B
39 v-I
40 vd-B
41 vd-I
42 vn-B
43 vn-I
44 w-B
45 w-I
46 xc-B
47 xc-I
48 PER-B
49 PER-I
50 LOC-B
51 LOC-I
52 ORG-B
53 ORG-I
54 TIME-B
55 TIME-I
56 O
demo/lac/resources/word.dic 已删除 100644 → 0
浏览文件 @ 2773c85b
此差异已折叠。
demo/senta/README.md
浏览文件 @ 2ae797eb
## 关于Senta
# senta
https://github.com/baidu/Senta
## 关于
本示例展示如何使用senta Module进行预测。
senta是中文情感分析模型，可以用于进行中文句子的情感分析，输出结果为`{正向/中性/负向}`中的一个，关于模型的训练细节，请查看[senta](https://github.com/baidu/senta)
## 准备工作
在运行本目录的脚本前，需要先安装1.3.0版本以上的PaddlePaddle（如果您本地已经安装了符合条件的PaddlePaddle版本，那么可以跳过`准备工作`这一步）。
如果您的机器支持GPU，我们建议下载GPU版本的PaddlePaddle，使用GPU进行训练和预测的效率都比使用CPU要高。
```shell
# 安装GPU版本的PaddlePaddle
$ pip install --upgrade paddlepaddle-gpu
```
如果您的机器不支持GPU，可以通过下面的命令来安装CPU版本的PaddlePaddle
## 创建Module
本目录包含了创建一个基于senta预训练模型的Module的脚本。
通过以下脚本来一键创建一个senta Module
```shell
$ sh create_module.sh
# 安装CPU版本的PaddlePaddle
$ pip install --upgrade paddlepaddle
```
NOTE:
* 如果进行下面示例的脚本或者代码，请确保执行上述脚本
* 关于创建Module的API和细节，请查看`create_module.py`
## 使用Module预测
该Module创建完成后，可以通过命令行或者python API两种方式进行预测
### 命令行方式
在安装过程中如果遇到问题，您可以到[Paddle官方网站](http://www.paddlepaddle.org/)上查看解决方案。
## 命令行方式预测
`infer.sh`给出了使用命令行调用Module预测的示例脚本
通过以下命令试验下效果
```shell
$ sh infer.sh
```
### 通过python API
## 通过python API预测
`infer_by_code.py`给出了使用python API调用Module预测的示例代码
通过以下命令试验下效果
```shell
$ python infer_by_code.py
python infer_by_code.py
```
demo/senta/create_module.py 已删除 100644 → 0
浏览文件 @ 2773c85b
import io
import paddle.fluid as fluid
import processor
import numpy as np
import nets
import paddlehub as hub
def load_vocab(file_path):
"""
load the given vocabulary
"""
vocab = {}
with io.open(file_path, 'r', encoding='utf8') as f:
wid = 0
for line in f:
line = line.rstrip()
parts = line.split('\t')
vocab[parts[0]] = int(parts[1])
vocab["<unk>"] = len(vocab)
return vocab
def create_module():
network = nets.bilstm_net
# word seq data
data = fluid.layers.data(
name="words", shape=[1], dtype="int64", lod_level=1)
# label data
label = fluid.layers.data(name="label", shape=[1], dtype="int64")
word_dict_path = "./resources/train.vocab"
word_dict = load_vocab(word_dict_path)
cost, acc, pred = network(data, label, len(word_dict) + 1)
place = fluid.CPUPlace()
exe = fluid.Executor(place)
model_path = "./resources/senta_model"
fluid.io.load_inference_model(model_path, exe)
# assets
assets = [word_dict_path]
# create a module
sign = hub.create_signature(
name="sentiment_classify",
inputs=[data],
outputs=[pred],
for_predict=True)
hub.create_module(
sign_arr=[sign],
module_dir="senta.hub_module",
exe=exe,
module_info="resources/module_info.yml",
processor=processor.Processor,
assets=assets)
if __name__ == "__main__":
create_module()
demo/senta/create_module.sh 已删除 100644 → 0
浏览文件 @ 2773c85b
#!/bin/bash
set -o nounset
set -o errexit
script_path=$(cd `dirname $0`; pwd)
module_path=senta.hub_module
if [ -d $script_path/$module_path ]
then
echo "$module_path already existed!"
exit 0
fi
cd $script_path/resources/
if [ ! -d senta_model ]
then
sh download.sh
fi
cd $script_path/
python create_module.py
echo "Successfully create $module_path"
demo/senta/infer.sh
浏览文件 @ 2ae797eb
python ../../paddlehub/commands/hub.py run senta.hub_module/ --signature sentiment_classify --config resources/test/test.yml --input_file resources/test/test.txt
python ../../paddlehub/commands/hub.py run senta --input_file test/test.txt
demo/senta/infer_by_code.py
浏览文件 @ 2ae797eb
......@@ -4,7 +4,7 @@ import paddlehub as hub
def infer_with_input_text():
# get senta module
senta = hub.Module(module_dir="senta.hub_module")
senta = hub.Module(name="senta")
test_text = ["这家餐厅很好吃", "这部电影真的很差劲"]
......@@ -24,14 +24,14 @@ def infer_with_input_text():
def infer_with_input_file():
# get senta module
senta = hub.Module(module_dir="senta.hub_module")
senta = hub.Module(name="senta")
# get the input keys for signature 'sentiment_classify'
data_format = senta.processor.data_format(sign_name='sentiment_classify')
key = list(data_format.keys())[0]
# parse input file
test_file = os.path.join("resources", "test", "test.txt")
test_file = os.path.join("test", "test.txt")
test_text = hub.io.parser.txt_parser.parse(test_file)
# set input dict
......
demo/senta/nets.py 已删除 100644 → 0
浏览文件 @ 2773c85b
import paddle.fluid as fluid
def bilstm_net(data,
label,
dict_dim,
emb_dim=128,
hid_dim=128,
hid_dim2=96,
class_dim=2,
emb_lr=30.0):
"""
Bi-Lstm net
"""
# embedding layer
emb = fluid.layers.embedding(
input=data,
size=[dict_dim, emb_dim],
param_attr=fluid.ParamAttr(learning_rate=emb_lr))
# bi-lstm layer
fc0 = fluid.layers.fc(input=emb, size=hid_dim * 4)
rfc0 = fluid.layers.fc(input=emb, size=hid_dim * 4)
lstm_h, c = fluid.layers.dynamic_lstm(
input=fc0, size=hid_dim * 4, is_reverse=False)
rlstm_h, c = fluid.layers.dynamic_lstm(
input=rfc0, size=hid_dim * 4, is_reverse=True)
# extract last layer
lstm_last = fluid.layers.sequence_last_step(input=lstm_h)
rlstm_last = fluid.layers.sequence_last_step(input=rlstm_h)
lstm_last_tanh = fluid.layers.tanh(lstm_last)
rlstm_last_tanh = fluid.layers.tanh(rlstm_last)
# concat layer
lstm_concat = fluid.layers.concat(input=[lstm_last, rlstm_last], axis=1)
# full connect layer
fc1 = fluid.layers.fc(input=lstm_concat, size=hid_dim2, act='tanh')
# softmax layer
prediction = fluid.layers.fc(input=fc1, size=class_dim, act='softmax')
cost = fluid.layers.cross_entropy(input=prediction, label=label)
avg_cost = fluid.layers.mean(x=cost)
acc = fluid.layers.accuracy(input=prediction, label=label)
return avg_cost, acc, prediction
demo/senta/processor.py 已删除 100644 → 0
浏览文件 @ 2773c85b
import os
import io
import paddle
import paddle.fluid as fluid
import numpy as np
import paddlehub as hub
def load_vocab(file_path):
"""
load the given vocabulary
"""
vocab = {}
with io.open(file_path, 'r', encoding='utf8') as f:
wid = 0
for line in f:
line = line.rstrip()
parts = line.split('\t')
vocab[parts[0]] = int(parts[1])
vocab["<unk>"] = len(vocab)
return vocab
def get_predict_label(pos_prob):
neg_prob = 1 - pos_prob
# threshold should be (1, 0.5)
neu_threshold = 0.55
if neg_prob > neu_threshold:
label, key = 0, "负面"
elif pos_prob > neu_threshold:
label, key = 2, "正面"
else:
label, key = 1, "中性"
return label, key
class Processor(hub.BaseProcessor):
def __init__(self, module):
self.module = module
assets_path = self.module.helper.assets_path()
word_dict_path = os.path.join(assets_path, "train.vocab")
self.word_dict = load_vocab(word_dict_path)
path = hub.default_module_manager.search_module("lac")
if path:
self.lac = hub.Module(module_dir=path)
else:
result, _, path = hub.default_module_manager.install_module("lac")
assert path, "can't found necessary module lac"
self.lac = hub.Module(module_dir=path)
def preprocess(self, sign_name, data_dict):
result = {'text': []}
processed = self.lac.lexical_analysis(data=data_dict)
unk_id = len(self.word_dict)
for index, data in enumerate(processed):
result_i = {'processed': []}
result_i['origin'] = data_dict['text'][index]
for word in data['word']:
if word in self.word_dict:
_index = self.word_dict[word]
else:
_index = unk_id
result_i['processed'].append(_index)
result['text'].append(result_i)
return result
def postprocess(self, sign_name, data_out, data_info, **kwargs):
if sign_name == "sentiment_classify":
result = []
pred = fluid.executor.as_numpy(data_out)
for index in range(len(data_info['text'])):
result_i = {}
result_i['text'] = data_info['text'][index]['origin']
label, key = get_predict_label(pred[0][index, 1])
result_i['sentiment_label'] = label
result_i['sentiment_key'] = key
result.append(result_i)
return result
def data_format(self, sign_name):
if sign_name == "sentiment_classify":
return {
"text": {
"type": hub.DataType.TEXT,
"feed_key": self.module.signatures[sign_name].inputs[0].name
}
}
return None
demo/senta/resources/download.sh 已删除 100644 → 0
浏览文件 @ 2773c85b
#!/bin/bash
set -o nounset
set -o errexit
script_path=$(cd `dirname $0`; pwd)
cd $script_path
wget --no-check-certificate https://paddlehub.bj.bcebos.com/paddle_model/senta.tar.gz
wget --no-check-certificate https://paddlehub.bj.bcebos.com/paddle_model/train.vocab
tar xvzf senta.tar.gz
rm senta.tar.gz
demo/senta/resources/module_info.yml 已删除 100644 → 0
浏览文件 @ 2773c85b
name: senta
type: nlp/sentiment_analysis
author: paddlepaddle
author_email: paddle-dev@baidu.com
summary: "Senta is a model used to analyse sentiment tendency of Chinese sentences. We divide sentiment tendencies into three levels, score 2 means positive, score 1 means neuter, and score 0 means negative"
version: 1.0.0
demo/sequence-labeling/predict.py
浏览文件 @ 2ae797eb
......@@ -53,29 +53,25 @@ if __name__ == '__main__':
place = fluid.CUDAPlace(0) if args.use_gpu else fluid.CPUPlace()
exe = fluid.Executor(place)
with fluid.program_guard(program):
label = fluid.layers.data(
name="label", shape=[args.max_seq_len, 1], dtype='int64')
seq_len = fluid.layers.data(name="seq_len", shape=[1], dtype='int64')
# Use "sequence_outputs" for token-level output.
sequence_output = output_dict["sequence_output"]
# Define a classfication finetune task by PaddleHub's API
seq_label_task = hub.create_seq_label_task(
feature=sequence_output,
num_classes=dataset.num_labels,
max_seq_len=args.max_seq_len)
# Setup feed list for data feeder
# Must feed all the tensor of ERNIE's module need
# Compared to classification task, we need add seq_len tensor to feedlist
feed_list = [
input_dict["input_ids"].name, input_dict["position_ids"].name,
input_dict["segment_ids"].name, input_dict["input_mask"].name,
label.name, seq_len
seq_label_task.variable('label').name,
seq_label_task.variable('seq_len').name
]
# Define a classfication finetune task by PaddleHub's API
seq_label_task = hub.create_seq_label_task(
feature=sequence_output,
labels=label,
num_classes=dataset.num_labels,
seq_len=seq_len)
fetch_list = [
seq_label_task.variable("labels").name,
seq_label_task.variable("infers").name,
......
demo/sequence-labeling/sequence_label.py
浏览文件 @ 2ae797eb
......@@ -69,7 +69,7 @@ if __name__ == '__main__':
strategy = hub.AdamWeightDecayStrategy(
weight_decay=args.weight_decay,
learning_rate=args.learning_rate,
lr_scheduler="linear_warmup_decay",
lr_scheduler="linear_decay",
)
# Setup runing config for PaddleHub Finetune API
......
demo/ssd/README.md
浏览文件 @ 2ae797eb
## 关于SSD
https://github.com/PaddlePaddle/models/tree/develop/fluid/PaddleCV/object_detection
# SSD
## 创建Module
本目录包含了创建一个基于POSCAL VOC数据集预训练的SSD模型的Module的脚本。
通过以下脚本来一键创建一个SSD Module
## 关于
本示例展示如何使用SSD Module进行预测。
SSD是一个目标检测模型，可以检测出图片中的实物的类别和位置，PaddleHub发布的SSD模型通过pascalvoc数据集训练，支持20个数据类别的检测，关于模型的训练细节，请查看[SSD](https://github.com/PaddlePaddle/models/tree/develop/PaddleCV/object_detection)
## 准备工作
在运行本目录的脚本前，需要先安装1.3.0版本以上的PaddlePaddle（如果您本地已经安装了符合条件的PaddlePaddle版本，那么可以跳过`准备工作`这一步）。
如果您的机器支持GPU，我们建议下载GPU版本的PaddlePaddle，使用GPU进行训练和预测的效率都比使用CPU要高。
```shell
sh create_module.sh
# 安装GPU版本的PaddlePaddle
$ pip install --upgrade paddlepaddle-gpu
```
NOTE:
* 如果进行下面示例的脚本或者代码，请确保执行上述脚本
* 关于创建Module的API和细节，请查看`create_module.py`
## 使用Module预测
该Module创建完成后，可以通过命令行或者python API两种方式进行预测
### 命令行方式
如果您的机器不支持GPU，可以通过下面的命令来安装CPU版本的PaddlePaddle
```shell
# 安装CPU版本的PaddlePaddle
$ pip install --upgrade paddlepaddle
```
在安装过程中如果遇到问题，您可以到[Paddle官方网站](http://www.paddlepaddle.org/)上查看解决方案。
## 命令行方式预测
`infer.sh`给出了使用命令行调用Module预测的示例脚本
通过以下命令试验下效果
```shell
sh infer.sh
$ sh infer.sh
```
### 通过python API
## 通过python API预测
`infer_by_code.py`给出了使用python API调用Module预测的示例代码
通过以下命令试验下效果
```shell
python infer_by_code.py
```
demo/ssd/create_module.py 已删除 100644 → 0
浏览文件 @ 2773c85b
import os
import numpy as np
import processor
import paddlehub as hub
import paddle
import paddle.fluid as fluid
from mobilenet_ssd import mobile_net
def build_program():
image_shape = [3, 300, 300]
class_num = 21
image = fluid.layers.data(dtype="float32", shape=image_shape, name="image")
gt_box = fluid.layers.data(
dtype="float32", shape=[4], name="gtbox", lod_level=1)
gt_label = fluid.layers.data(
dtype="int32", shape=[1], name="label", lod_level=1)
difficult = fluid.layers.data(
dtype="int32", shape=[1], name="difficult", lod_level=1)
with fluid.unique_name.guard():
locs, confs, box, box_var = mobile_net(class_num, image, image_shape)
nmsed_out = fluid.layers.detection_output(
locs, confs, box, box_var, nms_threshold=0.45)
return image, nmsed_out
def create_module():
image, nmsed_out = build_program()
place = fluid.CPUPlace()
exe = fluid.Executor(place)
exe.run(fluid.default_startup_program())
pretrained_model = "resources/ssd_mobilenet_v1_pascalvoc"
def if_exist(var):
return os.path.exists(os.path.join(pretrained_model, var.name))
fluid.io.load_vars(exe, pretrained_model, predicate=if_exist)
assets = ["resources/label_list.txt"]
sign = hub.create_signature(
"object_detection",
inputs=[image],
outputs=[nmsed_out],
for_predict=True)
hub.create_module(
sign_arr=[sign],
module_dir="ssd_mobilenet_v1_pascal.hub_module",
module_info="resources/module_info.yml",
exe=exe,
processor=processor.Processor,
assets=assets)
if __name__ == '__main__':
create_module()
demo/ssd/create_module.sh 已删除 100644 → 0
浏览文件 @ 2773c85b
#!/bin/bash
set -o nounset
set -o errexit
script_path=$(cd `dirname $0`; pwd)
module_path=ssd_mobilenet_v1_pascal.hub_module
if [ -d $script_path/$module_path ]
then
echo "$module_path already existed!"
exit 0
fi
cd $script_path/resources/
if [ ! -d ssd_mobilenet_v1_pascalvoc ]
then
sh download.sh
fi
cd $script_path/
python create_module.py
echo "Successfully create $module_path"
demo/ssd/infer.sh
浏览文件 @ 2ae797eb
python ../../paddlehub/commands/hub.py run ssd_mobilenet_v1_pascal.hub_module/ --signature object_detection --config resources/test/test.yml --input_file resources/test/test.txt
python ../../paddlehub/commands/hub.py run ssd_mobilenet_v1_pascal --input_file test/test.txt
demo/ssd/infer_by_code.py
浏览文件 @ 2ae797eb
......@@ -2,11 +2,11 @@ import os
import paddlehub as hub
def infer_with_input_text():
def infer_with_input_path():
# get ssd module
ssd = hub.Module(module_dir="ssd_mobilenet_v1_pascal.hub_module")
ssd = hub.Module(name="ssd_mobilenet_v1_pascal")
test_img_path = os.path.join("resources", "test", "test_img_bird.jpg")
test_img_path = os.path.join("test", "test_img_bird.jpg")
# get the input keys for signature 'object_detection'
data_format = ssd.processor.data_format(sign_name='object_detection')
......@@ -23,14 +23,14 @@ def infer_with_input_text():
def infer_with_input_file():
# get ssd module
ssd = hub.Module(module_dir="ssd_mobilenet_v1_pascal.hub_module")
ssd = hub.Module(name="ssd_mobilenet_v1_pascal")
# get the input keys for signature 'object_detection'
data_format = ssd.processor.data_format(sign_name='object_detection')
key = list(data_format.keys())[0]
# parse input file
test_file = os.path.join("resources", "test", "test.txt")
test_file = os.path.join("test", "test.txt")
test_images = hub.io.parser.txt_parser.parse(test_file)
# set input dict
......
demo/ssd/mobilenet_ssd.py 已删除 100644 → 0
浏览文件 @ 2773c85b
import paddle.fluid as fluid
from paddle.fluid.initializer import MSRA
from paddle.fluid.param_attr import ParamAttr
def conv_bn(input,
filter_size,
num_filters,
stride,
padding,
channels=None,
num_groups=1,
act='relu',
use_cudnn=True):
parameter_attr = ParamAttr(learning_rate=0.1, initializer=MSRA())
conv = fluid.layers.conv2d(
input=input,
num_filters=num_filters,
filter_size=filter_size,
stride=stride,
padding=padding,
groups=num_groups,
act=None,
use_cudnn=use_cudnn,
param_attr=parameter_attr,
bias_attr=False)
return fluid.layers.batch_norm(input=conv, act=act)
def depthwise_separable(input, num_filters1, num_filters2, num_groups, stride,
scale):
depthwise_conv = conv_bn(
input=input,
filter_size=3,
num_filters=int(num_filters1 * scale),
stride=stride,
padding=1,
num_groups=int(num_groups * scale),
use_cudnn=False)
pointwise_conv = conv_bn(
input=depthwise_conv,
filter_size=1,
num_filters=int(num_filters2 * scale),
stride=1,
padding=0)
return pointwise_conv
def extra_block(input, num_filters1, num_filters2, num_groups, stride, scale):
# 1x1 conv
pointwise_conv = conv_bn(
input=input,
filter_size=1,
num_filters=int(num_filters1 * scale),
stride=1,
num_groups=int(num_groups * scale),
padding=0)
# 3x3 conv
normal_conv = conv_bn(
input=pointwise_conv,
filter_size=3,
num_filters=int(num_filters2 * scale),
stride=2,
num_groups=int(num_groups * scale),
padding=1)
return normal_conv
def mobile_net(num_classes, img, img_shape, scale=1.0):
# 300x300
tmp = conv_bn(img, 3, int(32 * scale), 2, 1, 3)
# 150x150
tmp = depthwise_separable(tmp, 32, 64, 32, 1, scale)
tmp = depthwise_separable(tmp, 64, 128, 64, 2, scale)
# 75x75
tmp = depthwise_separable(tmp, 128, 128, 128, 1, scale)
tmp = depthwise_separable(tmp, 128, 256, 128, 2, scale)
# 38x38
tmp = depthwise_separable(tmp, 256, 256, 256, 1, scale)
tmp = depthwise_separable(tmp, 256, 512, 256, 2, scale)
# 19x19
for i in range(5):
tmp = depthwise_separable(tmp, 512, 512, 512, 1, scale)
module11 = tmp
tmp = depthwise_separable(tmp, 512, 1024, 512, 2, scale)
# 10x10
module13 = depthwise_separable(tmp, 1024, 1024, 1024, 1, scale)
module14 = extra_block(module13, 256, 512, 1, 2, scale)
# 5x5
module15 = extra_block(module14, 128, 256, 1, 2, scale)
# 3x3
module16 = extra_block(module15, 128, 256, 1, 2, scale)
# 2x2
module17 = extra_block(module16, 64, 128, 1, 2, scale)
mbox_locs, mbox_confs, box, box_var = fluid.layers.multi_box_head(
inputs=[module11, module13, module14, module15, module16, module17],
image=img,
num_classes=num_classes,
min_ratio=20,
max_ratio=90,
min_sizes=[60.0, 105.0, 150.0, 195.0, 240.0, 285.0],
max_sizes=[[], 150.0, 195.0, 240.0, 285.0, 300.0],
aspect_ratios=[[2.], [2., 3.], [2., 3.], [2., 3.], [2., 3.], [2., 3.]],
base_size=img_shape[2],
offset=0.5,
flip=True)
return mbox_locs, mbox_confs, box, box_var
demo/ssd/processor.py 已删除 100644 → 0
浏览文件 @ 2773c85b
import paddle
import paddlehub as hub
import numpy as np
import os
from paddlehub import BaseProcessor
from PIL import Image
from PIL import ImageDraw
from PIL import ImageFont
def clip_bbox(bbox):
xmin = max(min(bbox[0], 1.), 0.)
ymin = max(min(bbox[1], 1.), 0.)
xmax = max(min(bbox[2], 1.), 0.)
ymax = max(min(bbox[3], 1.), 0.)
return xmin, ymin, xmax, ymax
def draw_bounding_box_on_image(image_path, data_list, save_path):
image = Image.open(image_path)
draw = ImageDraw.Draw(image)
for data in data_list:
left, right, top, bottom = data['left'], data['right'], data[
'top'], data['bottom']
draw.line([(left, top), (left, bottom), (right, bottom), (right, top),
(left, top)],
width=4,
fill='red')
if image.mode == 'RGB':
draw.text((left, top), data['label'], (255, 255, 0))
image_name = os.path.split(image_path)[-1]
if not os.path.exists(save_path):
os.mkdir(save_path)
save_path = os.path.join(save_path, image_name)
print("image with bbox drawed saved as {}".format(
os.path.abspath(save_path)))
image.save(save_path)
class Processor(BaseProcessor):
def __init__(self, module):
self.module = module
label_list_file = os.path.join(self.module.helper.assets_path(),
"label_list.txt")
with open(label_list_file, "r") as file:
content = file.read()
self.label_list = content.split("\n")
self.confs_threshold = 0.5
def preprocess(self, sign_name, data_dict):
def process_image(img):
if img.mode == 'L':
img = im.convert('RGB')
im_width, im_height = img.size
img = img.resize((300, 300), Image.ANTIALIAS)
img = np.array(img)
# HWC to CHW
if len(img.shape) == 3:
img = np.swapaxes(img, 1, 2)
img = np.swapaxes(img, 1, 0)
# RBG to BGR
img = img[[2, 1, 0], :, :]
img = img.astype('float32')
mean_value = [127.5, 127.5, 127.5]
mean_value = np.array(mean_value)[:, np.newaxis, np.newaxis].astype(
'float32')
img -= mean_value
img = img * 0.007843
return img
result = {'image': []}
for path in data_dict['image']:
img = Image.open(path)
im_width, im_height = img.size
result_i = {}
result_i['path'] = path
result_i['width'] = im_width
result_i['height'] = im_height
result_i['processed'] = process_image(img)
result['image'].append(result_i)
return result
def postprocess(self, sign_name, data_out, data_info, **kwargs):
if sign_name == "object_detection":
lod_tensor = data_out[0]
lod = lod_tensor.lod()[0]
results = np.array(data_out[0])
output = []
for index in range(len(lod) - 1):
result_i = results[lod[index]:lod[index + 1]]
output_i = {
'path': data_info['image'][index]['path'],
'data': []
}
for dt in result_i:
if dt[1] < self.confs_threshold:
continue
dt_i = {}
category_id = dt[0]
bbox = dt[2:]
xmin, ymin, xmax, ymax = clip_bbox(dt[2:])
(left, right, top,
bottom) = (xmin * data_info['image'][index]['width'],
xmax * data_info['image'][index]['width'],
ymin * data_info['image'][index]['height'],
ymax * data_info['image'][index]['height'])
dt_i['left'] = left
dt_i['right'] = right
dt_i['top'] = top
dt_i['bottom'] = bottom
dt_i['label'] = self.label_list[int(category_id)]
output_i['data'].append(dt_i)
draw_bounding_box_on_image(
output_i['path'], output_i['data'], save_path="test_result")
output.append(output_i)
return output
def data_format(self, sign_name):
if sign_name == "object_detection":
return {
"image": {
'type': hub.DataType.IMAGE,
'feed_key': self.module.signatures[sign_name].inputs[0].name
}
}
return None
demo/ssd/resources/download.sh 已删除 100644 → 0
浏览文件 @ 2773c85b
#!/bin/bash
set -o nounset
set -o errexit
script_path=$(cd `dirname $0`; pwd)
cd $script_path
wget --no-check-certificate https://paddlehub.bj.bcebos.com/paddle_model/ssd_mobilenet_v1_pascalvoc.tar.gz
tar xvzf ssd_mobilenet_v1_pascalvoc.tar.gz
rm ssd_mobilenet_v1_pascalvoc.tar.gz
demo/ssd/resources/label_list.txt 已删除 100644 → 0
浏览文件 @ 2773c85b
background
aeroplane
bicycle
bird
boat
bottle
bus
car
cat
chair
cow
diningtable
dog
horse
motorbike
person
pottedplant
sheep
sofa
train
tvmonitor
demo/ssd/resources/module_info.yml 已删除 100644 → 0
浏览文件 @ 2773c85b
name: ssd_mobilenet_v1_pascalvoc
type: CV/object-detection
author: paddlepaddle
author_email: paddle-dev@baidu.com
summary: "SSD(Single Shot MultiBox Detector) is a object detection model use to detect target category and location in a image picture. This model is trained with PASCAL VOC dataset, and therefore provides 20 categories of recognition capability, which mentioned below : aeroplane,bicycle,bird,boat,bottle,bus,car,cat,chair,cow,diningtable,dog,horse,motorbike,person,pottedplant,sheep,sofa,train,tvmonitor"
version: 1.0.0
demo/ssd/resources/test/test.txt 已删除 100644 → 0
浏览文件 @ 2773c85b
./resources/test/test_img_sheep.jpg
./resources/test/test_img_cat.jpg
./resources/test/test_img_bird.jpg
demo/ssd/test/test.txt 0 → 100644
浏览文件 @ 2ae797eb
test/test_img_sheep.jpg
test/test_img_cat.jpg
test/test_img_bird.jpg
demo/text-classification/README.md
浏览文件 @ 2ae797eb
......@@ -111,7 +111,7 @@ strategy = hub.AdamWeightDecayStrategy(
learning_rate=5e-5,
weight_decay=0.01,
warmup_proportion=0.0,
lr_scheduler="linear_warmup_decay",
lr_scheduler="linear_decay",
)
config = hub.RunConfig(use_cuda=True, num_epoch=3, batch_size=32, strategy=strategy)
......@@ -124,7 +124,7 @@ hub.finetune_and_eval(task=cls_task, data_reader=reader, feed_list=feed_list, co
`learning_rate`: Finetune过程中的最大学习率;
`weight_decay`: 模型的正则项参数，默认0.01，如果模型有过拟合倾向，可适当调高这一参数;
`warmup_proportion`: 如果warmup_proportion>0, 例如0.1, 则学习率会在前10%的steps中线性增长至最高值learning_rate;
`lr_scheduler`: 有两种策略可选(1) `linear_warmup_decay`策略学习率会在最高点后以线性方式衰减; `noam_decay`策略学习率会在最高点以多项式形式衰减；
`lr_scheduler`: 有两种策略可选(1) `linear_decay`策略学习率会在最高点后以线性方式衰减; `noam_decay`策略学习率会在最高点以多项式形式衰减；
#### 运行配置
`RunConfig` 主要控制Finetune的训练，包含以下可控制的参数:
......
demo/text-classification/text_classifier.py
浏览文件 @ 2ae797eb
......@@ -78,7 +78,7 @@ if __name__ == '__main__':
strategy = hub.AdamWeightDecayStrategy(
weight_decay=args.weight_decay,
learning_rate=args.learning_rate,
lr_scheduler="linear_warmup_decay",
lr_scheduler="linear_decay",
)
# Setup runing config for PaddleHub Finetune API
......
docs/API/Strategy.md
浏览文件 @ 2ae797eb
......@@ -3,13 +3,13 @@
----
在PaddleHub中，Strategy代表了在对[Task](https://github.com/PaddlePaddle/PaddleHub/tree/develop/docs/API/Task.md)进行Finetune时，应该使用怎样的策略。这里的策略，包含了对预训练参数使用怎样的学习率，使用哪种类型的优化器，使用什么类型的正则化等
## `class paddlehub.finetune.strategy.AdamWeightDecayStrategy(learning_rate=1e-4, lr_scheduler="linear_warmup_decay", warmup_proportion=0.0, weight_decay=0.01, optimizer_name=None)`
## `class paddlehub.finetune.strategy.AdamWeightDecayStrategy(learning_rate=1e-4, lr_scheduler="linear_decay", warmup_proportion=0.0, weight_decay=0.01, optimizer_name=None)`
基于Adam优化器的学习率衰减策略
> ### 参数
> * learning_rate: 全局学习率。默认为1e-4
>
> * lr_scheduler: 学习率调度方法。默认为"linear_warmup_decay"
> * lr_scheduler: 学习率调度方法。默认为"linear_decay"
>
> * warmup_proportion: warmup所占比重
>
......
docs/command_line_introduction.md
浏览文件 @ 2ae797eb
......@@ -16,7 +16,7 @@ PaddleHub为Module/Model（关于Model和Module的区别，请查看下方的介
## `download`
用于下载百度提供的预训练Model
用于下载百度提供的Module/Model
`选项`
> `--output_path`：用于指定存放下载文件的目录，默认为当前目录
......
docs/turtorial/cv_finetune_turtorial.md
浏览文件 @ 2ae797eb
......@@ -47,14 +47,20 @@ import paddlehub as hub
import paddle.fluid as fluid
```
接下来我们要在PaddleHub中选择合适的预训练模型来Finetune，由于猫狗分类是一个图像分类任务，因此我们使用resnet50作为预训练模型（PaddleHub提供了丰富的预训练模型，我们建议您尝试不同的预训练模型来获得更好的性能）
接下来我们要在PaddleHub中选择合适的预训练模型来Finetune，由于猫狗分类是一个图像分类任务，因此我们使用经典的resnet50作为预训练模型（PaddleHub提供了丰富的预训练模型，我们建议您尝试不同的预训练模型来获得更好的性能）
```python
cv_classifer_module = hub.Module(name = "resnet_v2_50_imagenet")
# cv_classifer_module = hub.Module(name = "resnet_v2_101_imagenet")
# cv_classifer_module = hub.Module(name = "resnet_v2_152_imagenet")
# cv_classifer_module = hub.Module(name = "nasnet_imagenet")
# cv_classifer_module = hub.Module(name = "mobilenet_v2_imagenet")
module_map = {
"resnet50": "resnet_v2_50_imagenet",
"resnet101": "resnet_v2_101_imagenet",
"resnet152": "resnet_v2_152_imagenet",
"mobilenet": "mobilenet_v2_imagenet",
"nasnet": "nasnet_imagenet",
"pnasnet": "pnasnet_imagenet"
}
module_name = module_map["resnet50"]
cv_classifer_module = hub.Module(name = module_name)
```
## 三、数据准备
......@@ -62,15 +68,15 @@ cv_classifer_module = hub.Module(name = "resnet_v2_50_imagenet")
接着需要加载图片数据集。我们需要自己切分数据集，将数据集且分为训练集、验证集和测试集。
同时使用三个文本文件来记录对应的图片路径和标签
```
├─data: 数据目录
  ├─train_list.txt：训练集数据列表
  ├─test_list.txt：测试集数据列表
  ├─validate_list：验证集数据列表
  └─……
```
每个文件的格式如下
```text
```
图片1路径 图片1标签
图片2路径 图片2标签
……
......@@ -80,7 +86,7 @@ cv_classifer_module = hub.Module(name = "resnet_v2_50_imagenet")
```python
# 使用本地数据集
class mydataset(paddlehub.ImageClassificationDataset):
class mydataset(hub.ImageClassificationDataset):
self.base_path = "data"
self.train_list_file = "train_list.txt"
self.test_list_file = "test_list.txt"
......@@ -94,7 +100,7 @@ dataset = mydataset()
```python
# 直接用PaddleHub提供的数据集
dataset = paddlehub.dataset.dogcat()
dataset = hub.dataset.DogCat()
```
接着生成一个图像分类的reader，reader负责将dataset的数据进行预处理，接着以特定格式组织并输入给模型进行训练。
......@@ -123,13 +129,13 @@ data_reader = hub.reader.ImageClassificationReader(
```python
input_dict, output_dict, program = cv_classifer_module.context(trainable=True)
with fluid.program_guard(program):
label = fluid.layers.data(name="label", dtype="int64", shape=[1])
img = input_dict["image"]
feature_map = output_dict["feature_map"]
feed_list = [img.name, label.name]
task = hub.create_img_classification_task(
feature=feature_map, label=label, num_classes=dataset.num_labels)
img = input_dict["image"]
feature_map = output_dict["feature_map"]
task = hub.create_img_cls_task(
feature=feature_map, num_classes=dataset.num_labels)
feed_list = [img.name, task.variable("label").name]
```
## 五、选择运行时配置
......@@ -146,7 +152,7 @@ with fluid.program_guard(program):
`checkpoint_dir`：将训练的参数和数据保存到cv_finetune_turtorial_demo目录中
更多运行配置，请查看[RunConfig](https://github.com/PaddlePaddle/PaddleHub/tree/develop/docs/API.md)
更多运行配置，请查看[RunConfig](https://github.com/PaddlePaddle/PaddleHub/tree/develop/docs/API/RunConfig.md)
```python
config = hub.RunConfig(
......@@ -170,14 +176,69 @@ hub.finetune_and_eval(
训练过程中的性能数据会被记录到本地，我们可以通过visualdl来可视化这些数据
我们在shell中输入以下命令来启动visualdl
我们在shell中输入以下命令来启动visualdl，其中`${HOST_IP}`为本机IP，需要用户自行指定
```shell
$ visualdl --logdir ./cv_finetune_turtorial_demo --host ${HOST_IP}
$ visualdl --logdir ./cv_finetune_turtorial_demo --host ${HOST_IP} --port 8989
```
启动服务后，我们使用浏览器访问指定的url，可以看到训练以及预测的loss曲线和accuracy曲线
启动服务后，我们使用浏览器访问`${HOST_IP}:8989`，可以看到训练以及预测的loss曲线和accuracy曲线
![img](https://paddlehub.bj.bcebos.com/resources/cv_turtorial_vdl_log.JPG)
## 八、使用模型进行预测
当Finetune完成后，我们使用模型来进行预测，首先我们需要加载下训练好的参数
当Finetune完成后，我们使用模型来进行预测，整个预测流程大致可以分为以下几步：
1. 构建网络
2. 生成预测数据的Reader
3. 切换到预测的Program
4. 加载预训练好的参数
5. 运行Program进行预测
`注意`：预测所用的测试图片请自行准备
完整代码如下：
```python
import os
import numpy as np
def predict():
# Step 1: build Program
input_dict, output_dict, program = cv_classifer_module.context(trainable=True)
img = input_dict["image"]
feature_map = output_dict["feature_map"]
task = hub.create_img_cls_task(
feature=feature_map, num_classes=dataset.num_labels)
feed_list = [img.name]
# Step 2: create data reader
data = [
"test_img_dog.jpg",
"test_img_cat.jpg"
]
data_reader = hub.reader.ImageClassificationReader(
image_width=cv_classifer_module.get_expected_image_width(),
image_height=cv_classifer_module.get_expected_image_height(),
images_mean=cv_classifer_module.get_pretrained_images_mean(),
images_std=cv_classifer_module.get_pretrained_images_std(),
dataset=None)
predict_reader = data_reader.data_generator(
phase="predict", batch_size=1, data=data)
# Step 3: switch to inference program
with fluid.program_guard(task.inference_program()):
# Step 4: load pretrained parameters
place = fluid.CPUPlace()
exe = fluid.Executor(place)
pretrained_model_dir = os.path.join("cv_finetune_turtorial_demo", "best_model")
fluid.io.load_persistables(exe, pretrained_model_dir)
feeder = fluid.DataFeeder(feed_list=feed_list, place=place)
# Step 5: predict
for index, batch in enumerate(predict_reader()):
result, = exe.run(
feed=feeder.feed(batch), fetch_list=[task.variable('probs')])
predict_result = np.argsort(result[0])[::-1][0]
print("input %i is %s, and the predict result is %s" %
(index+1, data[index], predict_result))
predict()
```
paddlehub/dataset/__init__.py
浏览文件 @ 2ae797eb
......@@ -22,3 +22,6 @@ from .lcqmc import LCQMC
# CV Dataset
from .dogcat import DogCatDataset as DogCat
from .flowers import FlowersDataset as Flowers
from .stanford_dogs import StanfordDogsDataset as StanfordDogs
from .food101 import Food101Dataset as Food101
from .indoor67 import Indoor67Dataset as Indoor67
paddlehub/dataset/base_cv_dataset.py
浏览文件 @ 2ae797eb
......@@ -28,7 +28,9 @@ class ImageClassificationDataset(object):
self.train_list_file = None
self.test_list_file = None
self.validate_list_file = None
self.label_list_file = None
self.num_labels = 0
self.label_list = []
def _download_dataset(self, dataset_path, url):
if not os.path.exists(dataset_path):
......@@ -52,15 +54,16 @@ class ImageClassificationDataset(object):
break
line = line.strip()
items = line.split(" ")
if os.path.isabs(items[0]):
image_path = items[0]
if len(items) > 2:
image_path = " ".join(items[0:-1])
else:
if self.base_path is None:
image_path = items[0]
else:
image_path = os.path.join(self.base_path, items[0])
label = items[1]
data.append((image_path, items[1]))
image_path = items[0]
if not os.path.isabs(image_path):
if self.base_path is not None:
image_path = os.path.join(self.base_path,
image_path)
label = items[-1]
data.append((image_path, items[-1]))
if shuffle:
np.random.shuffle(data)
......@@ -70,6 +73,13 @@ class ImageClassificationDataset(object):
return _base_reader()
def label_dict(self):
if not self.label_list:
with open(os.path.join(self.base_path, self.label_list_file),
"r") as file:
self.label_list = file.read().split("\n")
return {index: key for index, key in enumerate(self.label_list)}
def train_data(self, shuffle=True):
train_data_path = os.path.join(self.base_path, self.train_list_file)
return self._parse_data(train_data_path, shuffle)
......
paddlehub/dataset/dogcat.py
浏览文件 @ 2ae797eb
......@@ -32,4 +32,5 @@ class DogCatDataset(ImageClassificationDataset):
self.train_list_file = "train_list.txt"
self.test_list_file = "test_list.txt"
self.validate_list_file = "validate_list.txt"
self.label_list_file = "label_list.txt"
self.num_labels = 2
paddlehub/dataset/flowers.py
浏览文件 @ 2ae797eb
......@@ -32,4 +32,5 @@ class FlowersDataset(ImageClassificationDataset):
self.train_list_file = "train_list.txt"
self.test_list_file = "test_list.txt"
self.validate_list_file = "validate_list.txt"
self.label_list_file = "label_list.txt"
self.num_labels = 5
paddlehub/dataset/food101.py 0 → 100644
浏览文件 @ 2ae797eb
# Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
import os
import paddlehub as hub
from paddlehub.dataset.base_cv_dataset import ImageClassificationDataset
class Food101Dataset(ImageClassificationDataset):
def __init__(self):
super(Food101Dataset, self).__init__()
dataset_path = os.path.join(hub.common.dir.DATA_HOME, "food-101",
"images")
self.base_path = self._download_dataset(
dataset_path=dataset_path,
url="https://paddlehub-dataset.bj.bcebos.com/Food101.tar.gz")
self.train_list_file = "train_list.txt"
self.test_list_file = "test_list.txt"
self.validate_list_file = "validate_list.txt"
self.label_list_file = "label_list.txt"
self.num_labels = 101
paddlehub/dataset/indoor67.py 0 → 100644
浏览文件 @ 2ae797eb
# Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
import os
import paddlehub as hub
from paddlehub.dataset.base_cv_dataset import ImageClassificationDataset
class Indoor67Dataset(ImageClassificationDataset):
def __init__(self):
super(Indoor67Dataset, self).__init__()
dataset_path = os.path.join(hub.common.dir.DATA_HOME, "Indoor67")
self.base_path = self._download_dataset(
dataset_path=dataset_path,
url="https://paddlehub-dataset.bj.bcebos.com/Indoor67.tar.gz")
self.train_list_file = "train_list.txt"
self.test_list_file = "test_list.txt"
self.validate_list_file = "validate_list.txt"
self.label_list_file = "label_list.txt"
self.num_labels = 67
paddlehub/dataset/stanford_dogs.py 0 → 100644
浏览文件 @ 2ae797eb
# Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
import os
import paddlehub as hub
from paddlehub.dataset.base_cv_dataset import ImageClassificationDataset
class StanfordDogsDataset(ImageClassificationDataset):
def __init__(self):
super(StanfordDogsDataset, self).__init__()
dataset_path = os.path.join(hub.common.dir.DATA_HOME,
"StanfordDogs-120")
self.base_path = self._download_dataset(
dataset_path=dataset_path,
url="https://paddlehub-dataset.bj.bcebos.com/StanfordDogs-120.tar.gz"
)
self.train_list_file = "train_list.txt"
self.test_list_file = "test_list.txt"
self.validate_list_file = "validate_list.txt"
self.label_list_file = "label_list.txt"
self.num_labels = 120
paddlehub/finetune/optimization.py
浏览文件 @ 2ae797eb
......@@ -27,18 +27,18 @@ def adam_weight_decay_optimization(loss,
learning_rate,
main_program,
weight_decay,
scheduler='linear_warmup_decay'):
scheduler='linear_decay'):
if warmup_steps > 0:
if scheduler == 'noam_decay':
scheduled_lr = fluid.layers.learning_rate_scheduler\
.noam_decay(1/(warmup_steps *(learning_rate ** 2)),
warmup_steps)
elif scheduler == 'linear_warmup_decay':
elif scheduler == 'linear_decay':
scheduled_lr = linear_warmup_decay(learning_rate, warmup_steps,
num_train_steps)
else:
raise ValueError("Unkown learning rate scheduler, should be "
"'noam_decay' or 'linear_warmup_decay'")
"'noam_decay' or 'linear_decay'")
optimizer = fluid.optimizer.Adam(learning_rate=scheduled_lr)
else:
optimizer = fluid.optimizer.Adam(learning_rate=learning_rate)
......
paddlehub/finetune/strategy.py
浏览文件 @ 2ae797eb
......@@ -44,12 +44,36 @@ class DefaultStrategy(object):
self._optimizer_name = optimizer_name
def execute(self, loss):
if self.optimizer.lower() == "adam":
self.optimizer = fluid.optimizer.Adam(
learning_rate=self.learning_rate)
elif self.optimizer.lower() == "sgd":
if self._optimizer_name.lower() == "sgd":
self.optimizer = fluid.optimizer.SGD(
learning_rate=self.learning_rate)
elif self._optimizer_name.lower() == "adagrad":
self.optimizer = fluid.optimizer.Adagrad(
learning_rate=self.learning_rate)
elif self._optimizer_name.lower() == "adamax":
self.optimizer = fluid.optimizer.Adamax(
learning_rate=self.learning_rate)
elif self._optimizer_name.lower() == "decayedadagrad":
self.optimizer = fluid.optimizer.DecayedAdagrad(
learning_rate=self.learning_rate)
elif self._optimizer_name.lower() == "ftrl":
self.optimizer = fluid.optimizer.Ftrl(
learning_rate=self.learning_rate)
elif self._optimizer_name.lower() == "larsmomentum":
self.optimizer = fluid.optimizer.LarsMomentum(
learning_rate=self.learning_rate)
elif self._optimizer_name.lower() == "momentum":
self.optimizer = fluid.optimizer.Momentum(
learning_rate=self.learning_rate)
elif self._optimizer_name.lower() == "decayedadagrad":
self.optimizer = fluid.optimizer.DecayedAdagrad(
learning_rate=self.learning_rate)
elif self._optimizer_name.lower() == "rmsprop":
self.optimizer = fluid.optimizer.RMSPropOptimizer(
learning_rate=self.learning_rate)
else:
self.optimizer = fluid.optimizer.Adam(
learning_rate=self.learning_rate)
if self.optimizer is not None:
self.optimizer.minimize(loss)
......@@ -64,14 +88,14 @@ class DefaultStrategy(object):
class AdamWeightDecayStrategy(DefaultStrategy):
def __init__(self,
learning_rate=1e-4,
lr_scheduler="linear_warmup_decay",
lr_scheduler="linear_decay",
warmup_proportion=0.0,
weight_decay=0.01,
optimizer_name=None):
super().__init__(
learning_rate=learning_rate, optimizer_name=optimizer_name)
# check strategy correctness
if lr_scheduler not in ["linear_warmup_decay", "noam_decay"]:
if lr_scheduler not in ["linear_decay", "noam_decay"]:
raise ValueError("lr_scheduler {} is not setup "
"correctly".format(lr_scheduler))
self._lr_scheduler = lr_scheduler
......@@ -129,12 +153,36 @@ class DefaultFinetuneStrategy(DefaultStrategy):
self.regularization_coeff = regularization_coeff
def execute(self, loss):
if self._optimizer_name.lower() == "adam":
self.optimizer = fluid.optimizer.Adam(
learning_rate=self.learning_rate)
elif self._optimizer_name.lower() == "sgd":
if self._optimizer_name.lower() == "sgd":
self.optimizer = fluid.optimizer.SGD(
learning_rate=self.learning_rate)
elif self._optimizer_name.lower() == "adagrad":
self.optimizer = fluid.optimizer.Adagrad(
learning_rate=self.learning_rate)
elif self._optimizer_name.lower() == "adamax":
self.optimizer = fluid.optimizer.Adamax(
learning_rate=self.learning_rate)
elif self._optimizer_name.lower() == "decayedadagrad":
self.optimizer = fluid.optimizer.DecayedAdagrad(
learning_rate=self.learning_rate)
elif self._optimizer_name.lower() == "ftrl":
self.optimizer = fluid.optimizer.Ftrl(
learning_rate=self.learning_rate)
elif self._optimizer_name.lower() == "larsmomentum":
self.optimizer = fluid.optimizer.LarsMomentum(
learning_rate=self.learning_rate)
elif self._optimizer_name.lower() == "momentum":
self.optimizer = fluid.optimizer.Momentum(
learning_rate=self.learning_rate)
elif self._optimizer_name.lower() == "decayedadagrad":
self.optimizer = fluid.optimizer.DecayedAdagrad(
learning_rate=self.learning_rate)
elif self._optimizer_name.lower() == "rmsprop":
self.optimizer = fluid.optimizer.RMSPropOptimizer(
learning_rate=self.learning_rate)
else:
self.optimizer = fluid.optimizer.Adam(
learning_rate=self.learning_rate)
# get pretrained parameters
program = loss.block.program
......
Markdown is supported
0% 或 .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑！
想要评论请 注册 或