未验证 提交 1088343d 编写于 作者: W Wu Sibo 提交者: GitHub

Merge branch 'develop' into dev/doc2

# Minimal makefile for Sphinx documentation
#
# You can set these variables from the command line, and also
# from the environment for the first two.
SPHINXOPTS ?=
SPHINXBUILD ?= sphinx-build
SOURCEDIR = source
BUILDDIR = build
# Put it first so that "make" without argument is like "make help".
help:
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
.PHONY: help Makefile
# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
## Contents
## Catalogue
- [1. Dataset Introduction](#1)
- [1.1 ImageNet-1k](#1.1)
......
......@@ -2,7 +2,7 @@
Experience the training, evaluation, and prediction of multi-label classification based on the [NUS-WIDE-SCENE](https://lms.comp.nus.edu.sg/wp-content/uploads/2019/research/nuswide/NUS-WIDE.html) dataset, which is a subset of the NUS-WIDE dataset. Please first install PaddlePaddle and PaddleClas, see [Paddle Installation](https://github.com/PaddlePaddle/PaddleClas/blob/develop/docs/zh_CN/installation) and [PaddleClas installation](https://github.com/PaddlePaddle/PaddleClas/blob/develop/docs/zh_CN/installation/install_ paddleclas.md) for more details.
## Contents
## Catalogue
- [1. Data and Model Preparation](#1)
- [2. Model Training](#2)
......@@ -114,4 +114,4 @@ Obtain an output silimar to the following:
```
0517_2715693311.jpg: class id(s): [6, 13, 17, 23, 26, 30], score(s): [0.96, 0.56, 0.55, 0.99, 0.59, 0.79], label_name(s): []
```
\ No newline at end of file
```
@ECHO OFF
pushd %~dp0
REM Command file for Sphinx documentation
if "%SPHINXBUILD%" == "" (
set SPHINXBUILD=sphinx-build
)
set SOURCEDIR=source
set BUILDDIR=build
if "%1" == "" goto help
%SPHINXBUILD% >NUL 2>NUL
if errorlevel 9009 (
echo.
echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
echo.installed, then set the SPHINXBUILD environment variable to point
echo.to the full path of the 'sphinx-build' executable. Alternatively you
echo.may add the Sphinx directory to PATH.
echo.
echo.If you don't have Sphinx installed, grab it from
echo.https://www.sphinx-doc.org/
exit /b 1
)
%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
goto end
:help
%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
:end
popd
.. paddleclas documentation master file, created by
sphinx-quickstart on Wed Dec 15 16:13:23 2021.
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
Welcome to paddleclas's documentation!
======================================
.. toctree::
:maxdepth: 2
:caption: Contents:
Indices and tables
==================
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`
# 数据增强
------
## 目录
- [1. 数据增强简介](#1)
......
......@@ -15,7 +15,7 @@
# sys.path.insert(0, os.path.abspath('.'))
import sphinx_rtd_theme
from recommonmark.parser import CommonMarkParser
#import sphinx-markdown-tables
import sphinx-markdown-tables
# -- Project information -----------------------------------------------------
project = 'PaddleClas'
......
......@@ -16,8 +16,4 @@
advanced_tutorials/index
others/index
faq_series/index
......@@ -7,6 +7,7 @@ REM Command file for Sphinx documentation
if "%SPHINXBUILD%" == "" (
set SPHINXBUILD=sphinx-build
)
set SOURCEDIR=source
set BUILDDIR=build
......
# 往期课程链接:
---
- [**【AI快车道PaddleClas系列直播课】**](https://aistudio.baidu.com/aistudio/course/introduce/24519)
- [图像识别系统解析](https://aistudio.baidu.com/aistudio/education/group/info/24519)
- 图像识别全能优势
- 整体架构及快速落地详解
- 个性化需求实现方案
- [商品识别系统全拆解](https://aistudio.baidu.com/aistudio/education/lessonvideo/1495317)
- 小样本多类别场景方案
- 图像检索技术及快速构建策略
- 动漫搜索趣味应用
- [车辆ReID核心技术方案](https://aistudio.baidu.com/aistudio/education/lessonvideo/1496537)
- ReID及跨境头场景应用
- Metric Learning——更鲁棒的检索特征
- Logo识别等方向延展
- [超轻量图像识别系统概览](https://aistudio.baidu.com/aistudio/education/lessonvideo/1890318)
- 图像识别技术选型策略
- 推理速度提升8倍的秘籍
- 四大典型行业应用案例
- [SOTA模型炼丹秘诀](https://aistudio.baidu.com/aistudio/education/lessonvideo/1890323)
- CPU定制模型PP-LCNet优化思路
- Vison Transformer模型的应用拓展
- [商品识别产业痛点剖析](https://aistudio.baidu.com/aistudio/education/lessonvideo/1896890)
- 特征提取技术详解
- 向量快速检索揭秘
- [手把手教你玩转图像识别](https://aistudio.baidu.com/aistudio/education/lessonvideo/1911507)
- 产业应用十问十答
- 智能零售下的应用案例
- 识别系统快速落地方案
......@@ -31,29 +31,29 @@
<a name='3'></a>
## 3. 使用 DALI
PaddleClas 支持在静态图训练方式中使用 DALI 加速,由于 DALI 仅支持 GPU 训练,因此需要设置 GPU,且 DALI 需要占用 GPU 显存,需要为 DALI 预留显存。使用 DALI 训练只需在训练配置文件中设置字段 `use_dali=True`,或通过以下命令启动训练即可:
PaddleClas 支持使用 DALI 对图像预处理进行加速,由于 DALI 仅支持 GPU 训练,因此需要设置 GPU,且 DALI 需要占用 GPU 显存,需要为 DALI 预留显存。使用 DALI 训练只需在训练配置文件中设置字段 `use_dali=True`,或通过以下命令启动训练即可:
```shell
# 设置用于训练的 GPU 卡号
export CUDA_VISIBLE_DEVICES="0"
python ppcls/static/train.py -c ppcls/configs/ImageNet/ResNet/ResNet50.yaml -o use_dali=True
python ppcls/train.py -c ppcls/configs/ImageNet/ResNet/ResNet50.yaml -o Global.use_dali=True
```
也可以使用多卡训练:
```shell
# 设置用于训练的 GPU 卡号
export CUDA_VISIBLE_DEVICES="0,1,2,3,4,5,6,7"
export CUDA_VISIBLE_DEVICES="0,1,2,3"
# 设置用于神经网络训练的显存大小,可根据具体情况设置,一般可设置为 0.8 或 0.7,剩余显存则预留 DALI 使用
export FLAGS_fraction_of_gpu_memory_to_use=0.80
python -m paddle.distributed.launch \
--gpus="0,1,2,3,4,5,6,7" \
ppcls/static/train.py \
--gpus="0,1,2,3" \
ppcls/train.py \
-c ./ppcls/configs/ImageNet/ResNet/ResNet50.yaml \
-o use_dali=True
-o Global.use_dali=True
```
<a name='4'></a>
......@@ -62,11 +62,11 @@ python -m paddle.distributed.launch \
在上述基础上,使用 FP16 半精度训练,可以进一步提高速度,可以参考下面的配置与运行命令。
```shell
export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
export CUDA_VISIBLE_DEVICES=0,1,2,3
export FLAGS_fraction_of_gpu_memory_to_use=0.8
python -m paddle.distributed.launch \
--gpus="0,1,2,3,4,5,6,7" \
ppcls/static/train.py \
-c ./ppcls/configs/ImageNet/ResNet/ResNet50_fp16.yaml
--gpus="0,1,2,3" \
ppcls/train.py \
-c ./ppcls/configs/ImageNet/ResNet/ResNet50_fp16_dygraph.yaml
```
# global configs
Global:
checkpoints: null
pretrained_model: https://paddle-imagenet-models-name.bj.bcebos.com/dygraph/rec/models/pretrain/general_PPLCNet_x2_5_pretrained_v1.0.pdparams
output_dir: ./output/
device: gpu
save_interval: 1
eval_during_train: True
eval_interval: 1
epochs: 30
print_batch_step: 10
use_visualdl: False
# used for static mode and model export
image_shape: [3, 224, 224]
save_inference_dir: ./inference
eval_mode: retrieval
use_dali: False
to_static: False
# for quantizaiton or prune model
Slim:
## for prune
quant:
name: pact
# model architecture
Arch:
name: RecModel
infer_output_key: features
infer_add_softmax: False
Backbone:
name: PPLCNet_x2_5
pretrained: False
use_ssld: True
BackboneStopLayer:
name: flatten_0
Neck:
name: FC
embedding_size: 1280
class_num: 512
Head:
name: ArcMargin
embedding_size: 512
class_num: 185341
margin: 0.2
scale: 30
# loss function config for traing/eval process
Loss:
Train:
- CELoss:
weight: 1.0
Eval:
- CELoss:
weight: 1.0
Optimizer:
name: Momentum
momentum: 0.9
lr:
name: Cosine
learning_rate: 0.002
warmup_epoch: 5
regularizer:
name: 'L2'
coeff: 0.00001
# data loader for train and eval
DataLoader:
Train:
dataset:
name: ImageNetDataset
image_root: ./dataset/
cls_label_path: ./dataset/train_reg_all_data.txt
transform_ops:
- DecodeImage:
to_rgb: True
channel_first: False
- RandCropImage:
size: 224
- RandFlipImage:
flip_code: 1
- NormalizeImage:
scale: 1.0/255.0
mean: [0.485, 0.456, 0.406]
std: [0.229, 0.224, 0.225]
order: ''
sampler:
name: DistributedBatchSampler
batch_size: 128
drop_last: False
shuffle: True
loader:
num_workers: 4
use_shared_memory: True
Eval:
Query:
dataset:
name: VeriWild
image_root: ./dataset/Aliproduct/
cls_label_path: ./dataset/Aliproduct/val_list.txt
transform_ops:
- DecodeImage:
to_rgb: True
channel_first: False
- ResizeImage:
size: 224
- NormalizeImage:
scale: 0.00392157
mean: [0.485, 0.456, 0.406]
std: [0.229, 0.224, 0.225]
order: ''
sampler:
name: DistributedBatchSampler
batch_size: 64
drop_last: False
shuffle: False
loader:
num_workers: 4
use_shared_memory: True
Gallery:
dataset:
name: VeriWild
image_root: ./dataset/Aliproduct/
cls_label_path: ./dataset/Aliproduct/val_list.txt
transform_ops:
- DecodeImage:
to_rgb: True
channel_first: False
- ResizeImage:
size: 224
- NormalizeImage:
scale: 0.00392157
mean: [0.485, 0.456, 0.406]
std: [0.229, 0.224, 0.225]
order: ''
sampler:
name: DistributedBatchSampler
batch_size: 64
drop_last: False
shuffle: False
loader:
num_workers: 4
use_shared_memory: True
Metric:
Eval:
- Recallk:
topk: [1, 5]
# global configs
Global:
checkpoints: null
pretrained_model: null
output_dir: ./output/
device: gpu
save_interval: 1
eval_during_train: True
eval_interval: 1
epochs: 60
print_batch_step: 10
use_visualdl: False
# used for static mode and model export
image_shape: [3, 224, 224]
save_inference_dir: ./inference
# for quantalization or prune model
Slim:
## for quantization
quant:
name: pact
# model architecture
Arch:
name: PPLCNet_x1_0
class_num: 1000
pretrained: True
# loss function config for traing/eval process
Loss:
Train:
- CELoss:
weight: 1.0
epsilon: 0.1
Eval:
- CELoss:
weight: 1.0
Optimizer:
name: Momentum
momentum: 0.9
lr:
name: Cosine
learning_rate: 0.02
warmup_epoch: 0
regularizer:
name: 'L2'
coeff: 0.00003
# data loader for train and eval
DataLoader:
Train:
dataset:
name: ImageNetDataset
image_root: ./dataset/ILSVRC2012/
cls_label_path: ./dataset/ILSVRC2012/train_list.txt
transform_ops:
- DecodeImage:
to_rgb: True
channel_first: False
- RandCropImage:
size: 224
- RandFlipImage:
flip_code: 1
- AutoAugment:
- NormalizeImage:
scale: 1.0/255.0
mean: [0.485, 0.456, 0.406]
std: [0.229, 0.224, 0.225]
order: ''
sampler:
name: DistributedBatchSampler
batch_size: 128
drop_last: False
shuffle: True
loader:
num_workers: 4
use_shared_memory: True
Eval:
dataset:
name: ImageNetDataset
image_root: ./dataset/ILSVRC2012/
cls_label_path: ./dataset/ILSVRC2012/val_list.txt
transform_ops:
- DecodeImage:
to_rgb: True
channel_first: False
- ResizeImage:
resize_short: 256
- CropImage:
size: 224
- NormalizeImage:
scale: 1.0/255.0
mean: [0.485, 0.456, 0.406]
std: [0.229, 0.224, 0.225]
order: ''
sampler:
name: DistributedBatchSampler
batch_size: 64
drop_last: False
shuffle: False
loader:
num_workers: 4
use_shared_memory: True
Infer:
infer_imgs: docs/images/whl/demo.jpg
batch_size: 10
transforms:
- DecodeImage:
to_rgb: True
channel_first: False
- ResizeImage:
resize_short: 256
- CropImage:
size: 224
- NormalizeImage:
scale: 1.0/255.0
mean: [0.485, 0.456, 0.406]
std: [0.229, 0.224, 0.225]
order: ''
- ToCHWImage:
PostProcess:
name: Topk
topk: 5
class_id_map_file: ppcls/utils/imagenet1k_label_list.txt
Metric:
Train:
- TopkAcc:
topk: [1, 5]
Eval:
- TopkAcc:
topk: [1, 5]
......@@ -250,6 +250,8 @@ class Engine(object):
self.scaler = paddle.amp.GradScaler(
init_loss_scaling=self.scale_loss,
use_dynamic_loss_scaling=self.use_dynamic_loss_scaling)
if self.config['AMP']['use_pure_fp16'] is True:
self.model = paddle.amp.decorate(models=self.model, level='O2')
self.max_iter = len(self.train_dataloader) - 1 if platform.system(
) == "Windows" else len(self.train_dataloader)
......
......@@ -21,6 +21,7 @@ from ppcls.utils import profiler
def train_epoch(engine, epoch_id, print_batch_step):
tic = time.time()
v_current = [int(i) for i in paddle.__version__.split(".")]
for iter_id, batch in enumerate(engine.train_dataloader):
if iter_id >= engine.max_iter:
break
......@@ -41,14 +42,15 @@ def train_epoch(engine, epoch_id, print_batch_step):
# image input
if engine.amp:
with paddle.amp.auto_cast(custom_black_list={
"flatten_contiguous_range", "greater_than"
}):
amp_level = 'O1'
if engine.config['AMP']['use_pure_fp16'] is True:
amp_level = 'O2'
with paddle.amp.auto_cast(custom_black_list={"flatten_contiguous_range", "greater_than"}, level=amp_level):
out = forward(engine, batch)
loss_dict = engine.train_loss_func(out, batch[1])
else:
out = forward(engine, batch)
loss_dict = engine.train_loss_func(out, batch[1])
loss_dict = engine.train_loss_func(out, batch[1])
# step opt and lr
if engine.amp:
......
......@@ -17,6 +17,7 @@ from __future__ import division
from __future__ import print_function
from paddle import optimizer as optim
import paddle
from ppcls.utils import logger
......@@ -36,7 +37,7 @@ class Momentum(object):
momentum,
weight_decay=None,
grad_clip=None,
multi_precision=False):
multi_precision=True):
super().__init__()
self.learning_rate = learning_rate
self.momentum = momentum
......@@ -55,6 +56,15 @@ class Momentum(object):
grad_clip=self.grad_clip,
multi_precision=self.multi_precision,
parameters=parameters)
if hasattr(opt, '_use_multi_tensor'):
opt = optim.Momentum(
learning_rate=self.learning_rate,
momentum=self.momentum,
weight_decay=self.weight_decay,
grad_clip=self.grad_clip,
multi_precision=self.multi_precision,
parameters=parameters,
use_multi_tensor=True)
return opt
......
1. 注册ReadtheDocs并连接到github
2. 在github上将项目克隆到本地
3. 在本地仓库中安装Sphinx
```shell
pip install sphinx
```
4. 创建工程
```shell
sphinx-quickstart
```
5. 对工程进行配置
5.1 更改主题
在source/conf.py中更改或添加如下代码
```python
import sphinx_rtd_theme
html_theme = "sphinx_rtd_theme"
html_theme_path = [sphinx_rtd_theme.get_html_theme_path()]
```
5.2 添加markdown支持和markdown表格支持
首先需要安装recommonmark和sphinx_markdown_tables
```shell
pip install recommonmark
pip install sphinx_markdown_tables
```
在source/conf.py中更改或添加如下代码
```python
from recommonmark.parser import CommonMarkParser
source_parsers = {
'.md': CommonMarkParser,
}
source_suffix = ['.rst', '.md']
extensions = [
'recommonmark',
'sphinx_markdown_tables'
]
```
以上五步具体效果可以参考https://www.jianshu.com/p/d1d59d0cd58c
6. 在创建好项目以后,根目录下应该有如下几个文件:
- **Makefile**:在使用 `make` 命令时,可以使用这些指令(e.g. `sphinx-build`)来构建文档输出。
- **_build**:这是触发特定输出后用来存放所生成的文件的目录。
- **_static**:所有不属于源代码(e.g. 图片)一部分的文件均存放于此处,稍后会在构建目录中将它们链接在一起。
- **conf.py**:用于存放 Sphinx 的配置值,包括在终端执行 `sphinx-quickstart`时选中的那些值。
- **index.rst**:文档项目的 root 目录。如果将文档划分为其他文件,该目录会连接这些文件
7. **编写文档**:在 index.rst 文件中的主标题之后,有一个内容清单,其中包括 `toctree` 声明,它将所有文档链接都汇集到 Index。
以根目录下的index.rst为例:
```rst
欢迎使用PaddleClas图像分类库!
================================
.. toctree::
:maxdepth: 1
models_training/index
introduction/index
image_recognition_pipeline/index
others/index
faq_series/index
data_preparation/index
installation/index
models/index
advanced_tutorials/index
algorithm_introduction/index
inference_deployment/index
quick_start/index
```
可以用下面的python代码实现根目录和各个子目录下的`index.rst`文件的编写
注意:此代码应该在需要生成文档书的文件夹根目录上运行
```python
import os
def file_name(file_dir):
temp = []
for root, dirs, files in os.walk(file_dir):
print(dirs) #当前路径下所有子目录
temp = dirs #存储需要的子目录
break
# 删除不需要的子目录
temp.remove('images')
temp.remove('_templates')
temp.remove('_build')
temp.remove('_static')
chinese_name = ['模型训练', '介绍', '图像识别流程', '其他', 'FAQ系列', '数据准备', '安装', '模型库', '高级教程', '算法介绍', '推理部署', '快速开始']
# 写根目录下的rst文件
with open('./index.rst', 'w') as f:
f.write('欢迎使用PaddleClas图像分类库!\n')
f.write('================================\n\n')
f.write('.. toctree::\n')
f.write(' :maxdepth: 1\n\n')
for dir in temp:
f.write(' ' + dir + '/index\n')
f.close()
# 写各个子目录下的rst文件
for dir in temp:
for root, dirs, files in os.walk(dir):
print(root) #当前目录路径
files.remove('index.rst')
print(files) #当前路径下所有非目录子文件
curDir = os.path.join(file_dir, dir)
filename = curDir + '/index.rst'
idx = temp.index(dir)
ch_name = chinese_name[idx]
with open(filename, 'w') as f:
f.write(ch_name+'\n')
f.write('================================\n\n')
f.write('.. toctree::\n')
f.write(' :maxdepth: 2\n\n')
for f1 in files:
f.write(' ' + f1 + '\n')
f.close()
def readfile(filename):
file = open(filename)
i = 0
while 1:
line = file.readline()
print(i)
print(line)
i += 1
if not line:
break
pass # do something
file.close()
file_name('./')
# filename = './index.rst'
# readfile(filename)
```
8. 生成文档
运行 `make html` 命令
9. 使用浏览器查看在build/html目录下的 `index.html`文件可以查看静态网页
......@@ -6,7 +6,7 @@ gpu_list:0|0,1
-o Global.auto_cast:null
-o Global.epochs:lite_train_lite_infer=2|whole_train_whole_infer=120
-o Global.output_dir:./output/
-o DataLoader.Train.sampler.batch_size:8
-o DataLoader.Train.sampler.batch_size:2
-o Global.pretrained_model:null
train_model_name:latest
train_infer_img_dir:./dataset/ILSVRC2012/val
......@@ -37,7 +37,7 @@ pretrained_model_url:https://paddle-imagenet-models-name.bj.bcebos.com/dygraph/L
infer_model:../inference/
infer_export:True
infer_quant:Fasle
inference:python/predict_cls.py -c configs/inference_cls.yaml -o PreProcess.transform_ops.0.ResizeImage.resize_short=384 -o PreProcess.transform_ops.1.CropImage.size=384
inference:python/predict_cls.py -c configs/inference_cls.yaml -o PreProcess.transform_ops.0.ResizeImage.resize_short=256 -o PreProcess.transform_ops.1.CropImage.size=224
-o Global.use_gpu:True|False
-o Global.enable_mkldnn:True|False
-o Global.cpu_num_threads:1|6
......
......@@ -6,7 +6,7 @@ gpu_list:0|0,1
-o Global.auto_cast:null
-o Global.epochs:lite_train_lite_infer=2|whole_train_whole_infer=120
-o Global.output_dir:./output/
-o DataLoader.Train.sampler.batch_size:8
-o DataLoader.Train.sampler.batch_size:2
-o Global.pretrained_model:null
train_model_name:latest
train_infer_img_dir:./dataset/ILSVRC2012/val
......
......@@ -6,7 +6,7 @@ gpu_list:0|0,1
-o Global.auto_cast:null
-o Global.epochs:lite_train_lite_infer=2|whole_train_whole_infer=120
-o Global.output_dir:./output/
-o DataLoader.Train.sampler.batch_size:8
-o DataLoader.Train.sampler.batch_size:2
-o Global.pretrained_model:null
train_model_name:latest
train_infer_img_dir:./dataset/ILSVRC2012/val
......
Markdown is supported
0% .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑!
想要评论请 注册