未验证 提交 b861c99a 编写于 作者: littletomatodonkey's avatar littletomatodonkey 提交者: GitHub

polish template (#5369)

上级 674dfa3f
#!/usr/bin/env bash
set -e
readonly VERSION="3.8"
version=$(clang-format -version)
if ! [[ $version == *"$VERSION"* ]]; then
echo "clang-format version check failed."
echo "a version contains '$VERSION' is needed, but get '$version'"
echo "you can install the right version, and make an soft-link to '\$PATH' env"
exit -1
fi
clang-format $@
[submodule "PaddleRL"]
path = PaddleRL
url = https://github.com/PaddlePaddle/PARL
[submodule "PaddleCV/PaddleSeg"]
path = PaddleCV/PaddleSeg
url = https://github.com/PaddlePaddle/PaddleSeg.git
[submodule "AutoDL"]
path = AutoDL
url = https://github.com/PaddlePaddle/AutoDL.git
[submodule "PaddleSpeech/DeepSpeech"]
path = PaddleSpeech/DeepSpeech
url = https://github.com/PaddlePaddle/DeepSpeech.git
[submodule "PaddleSpeech/Parakeet"]
path = PaddleSpeech/Parakeet
url = https://github.com/PaddlePaddle/Parakeet
[submodule "PaddleCV/PaddleDetection"]
path = PaddleCV/PaddleDetection
url = https://github.com/PaddlePaddle/PaddleDetection.git
[style]
based_on_style = pep8
column_limit = 80
language: cpp
cache: ccache
sudo: required
dist: trusty
services:
- docker
os:
- linux
env:
- JOB=PRE_COMMIT
addons:
apt:
packages:
- git
- python
- python-pip
- python2.7-dev
- clang-format-3.8
ssh_known_hosts: 13.229.163.131
before_install:
- if [[ "$JOB" == "PRE_COMMIT" ]]; then sudo ln -s /usr/bin/clang-format-3.8 /usr/bin/clang-format; fi
- sudo pip install -U virtualenv pre-commit pip
- docker pull paddlepaddle/paddle:latest
script:
- exit_code=0
- .travis/precommit.sh || exit_code=$(( exit_code | $? ))
- docker run -i --rm -v "$PWD:/py_unittest" paddlepaddle/paddle:latest /bin/bash -c
'cd /py_unittest; sh .travis/unittest.sh' || exit_code=$(( exit_code | $? ))
- |
script/deploy_doc.sh
notifications:
email:
on_success: change
on_failure: always
# REPO 提交规范
项目和代码的规范和可读性对于项目开发至关重要,可以提升开发效率。本文给出开发者在新建开发者生态项目时的repo目录示例,以供参考。
本文示例项目在文件夹[repo_template](./repo_template)下,您可以将这个文件夹中的内容拷贝出去,放在自己的项目文件夹下,并编写对应的代码与文档。
## 1. 目录结构
建议的目录结构如下:
```
./repo_template # 项目文件夹名称,可以修改为自己的文件夹名称
|-- config # 参数配置文件夹
|-- dataset # 数据处理代码文件夹
|-- images # 测试图片文件夹
|-- model # 模型实现文件夹
|-- utils # 功能类API文件夹
|-- deploy # 预测部署相关
| ├── pdinference # 基于PaddleInference的python推理代码文件夹
| ├── pdserving # 基于PaddleServing的推理代码文件夹
|-- tools # 工具类文件夹
| ├── train.py # 训练代码文件
| ├── eval.py # 评估代码文件
| ├── infer.py # 预测代码文件
| ├── export.py # 模型导出代码文件
|-- scripts # 脚本类文件夹
| ├── train.sh # 训练脚本,需要包含单机单卡和单机多卡训练的方式,单机多卡的训练方式可以以注释的形式给出
| ├── eval.sh # 评估脚本,提供单机单卡的评估方式即可
| ├── infer.sh # 预测脚本
| ├── export.sh # 模型导出脚本
|-- test_tipc # 训推一体测试文件夹
|-- README_en.md # 英文用户手册
|-- README.md # 中文用户手册
|-- LICENSE # LICENSE文件
```
- **config:** 存储模型配置相关文件的文件夹,保存模型的配置信息,如 `configs.py、configs.yml`
- **dataset:** 存储数据相关文件的文件夹,包含数据下载、数据处理等,如 `dataset_download.py、dataset_process.py`
- **images:** 存储项目相关的图片,首页以及TIPC文档中需要的图像都可以放在这里,如果需要进一步区分功能,可以在里面建立不同的子文件夹。
- **model:** 存储模型相关代码文件的文件夹,保存模型的实现,如 `resnet.py、cyclegan.py`
- **utils:** 存储功能类相关文件的文件夹,如可视化,文件夹操作,模型保存与加载等
- **deploy:** 部署相关文件夹,目前包含PaddleInference推理文件夹以及PaddleServing服务部署文件夹
- **tools:** 工具类文件夹,包含训练、评估、预测、模型导出等代码文件
- **scripts:** 工具类文件夹,包含训练、评估、预测、模型导出等脚本文件
- **test_tipc:** 训练一体 (TIPC) 测试文件夹
- **README_en.md:** 中文版当前模型的使用说明,规范参考 README 内容要求
- **README.md:** 英文版当前模型的使用说明,规范参考 README 内容要求
- **LICENSE:** LICENSE文件
## 2. 功能实现
模型需要提供的功能包含:
- 训练:可以在GPU单机单卡、单机多卡、CPU多核的环境下执行训练
- 预测:可以在GPU单卡和CPU单核下执行预测
- 评估:可以在GPU单卡和CPU单核下执行评估
- 模型导出:可以导出inference模型,并且跑通PaddleInference推理以及PaddleServing部署
- 使用自定义数据:要求模型可以灵活支持/适配自定义数据,可以通过在README中加入数据格式描部分和如何使用自定义数据章节解决
### 3. 命名规范和使用规范
- 文件和文件夹命名中,尽量使用下划线`_`代表空格,不要使用`-`
- 模型定义过程中,需要有一个统一的变量(parameter)命名管理手段,如尽量手动声明每个变量的名字并支持名称可变,禁止将名称定义为一个常数(如"embedding"),避免在复用代码阶段出现各种诡异的问题。
- 重要文件,变量的名称定义过程中需要能够通过名字表明含义,禁止使用含混不清的名称,如net.py, aaa.py等。
- 在代码中定义path时,需要使用os.path.join完成,禁止使用string加的方式,导致模型对windows环境缺乏支持。
### 4. 注释和License
对于代码中重要的部分,需要加入注释介绍功能,帮助用户快速熟悉代码结构,包括但不仅限于:
- Dataset、DataLoader的定义。
- 整个模型定义,包括input,运算过程,loss等内容。
- init,save,load,等io部分
- 运行中间的关键状态,如print loss,save model等。
如:
```
import random
from paddle.io import Dataset
from paddle.vision.transforms import transforms as T
class PetDataset(Dataset):
"""
Pet 数据集定义
"""
def __init__(self, mode='train'):
"""
构造函数
"""
self.image_size = IMAGE_SIZE
self.mode = mode.lower()
assert self.mode in ['train', 'test', 'predict'], \
"mode should be 'train' or 'test' or 'predict', but got {}".format(self.mode)
self.train_images = []
self.label_images = []
with open('./{}.txt'.format(self.mode), 'r') as f:
for line in f.readlines():
image, label = line.strip().split('\t')
self.train_images.append(image)
self.label_images.append(label)
def _load_img(self, path, color_mode='rgb', transforms=[]):
"""
统一的图像处理接口封装,用于规整图像大小和通道
"""
with open(path, 'rb') as f:
img = PilImage.open(io.BytesIO(f.read()))
if color_mode == 'grayscale':
# if image is not already an 8-bit, 16-bit or 32-bit grayscale image
# convert it to an 8-bit grayscale image.
if img.mode not in ('L', 'I;16', 'I'):
img = img.convert('L')
elif color_mode == 'rgba':
if img.mode != 'RGBA':
img = img.convert('RGBA')
elif color_mode == 'rgb':
if img.mode != 'RGB':
img = img.convert('RGB')
else:
raise ValueError('color_mode must be "grayscale", "rgb", or "rgba"')
return T.Compose([
T.Resize(self.image_size)
] + transforms)(img)
def __getitem__(self, idx):
"""
返回 image, label
"""
train_image = self._load_img(self.train_images[idx],
transforms=[
T.Transpose(),
T.Normalize(mean=127.5, std=127.5)
]) # 加载原始图像
label_image = self._load_img(self.label_images[idx],
color_mode='grayscale',
transforms=[T.Grayscale()]) # 加载Label图像
# 返回image, label
train_image = np.array(train_image, dtype='float32')
label_image = np.array(label_image, dtype='int64')
return train_image, label_image
def __len__(self):
"""
返回数据集总数
"""
return len(self.train_images)
```
对于整个模型代码,都需要在文件头内加入licenses,readme中加入licenses标识。
文件头内licenses格式如下:
```
#encoding=utf8
# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
# http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
```
## 5. 其他问题
- 使用 Paddle 2.x API开发,不使用 `paddle.fluid.*` 下的API;
- 代码封装得当,易读性好,不用一些随意的变量/类/函数命名
- 注释清晰,不仅说明做了什么,也要说明为什么这么做
- 如果模型依赖paddlepaddle未涵盖的依赖(如 pandas),则需要在README中显示提示用户安装对应依赖
- 随机控制,需要尽量固定含有随机因素模块的随机种子,保证模型可以正常复现
- 超参数:模型内部超参数禁止写死,尽量都可以通过配置文件进行配置。
## 6. README 内容&格式说明
模型的readme共分为以下几个部分,**具体模板见:**[README.md](repo_template/README.md)
```
# 模型名称
## 1. 简介
## 2. 复现精度
## 3. 数据集
## 4. 环境依赖
## 5. 快速开始
## 6. 代码结构与详细说明
## 7. 模型信息
```
# This file is used by clang-format to autoformat paddle source code
#
# The clang-format is part of llvm toolchain.
# It need to install llvm and clang to format source code style.
#
# The basic usage is,
# clang-format -i -style=file PATH/TO/SOURCE/CODE
#
# The -style=file implicit use ".clang-format" file located in one of
# parent directory.
# The -i means inplace change.
#
# The document of clang-format is
# http://clang.llvm.org/docs/ClangFormat.html
# http://clang.llvm.org/docs/ClangFormatStyleOptions.html
---
Language: Cpp
BasedOnStyle: Google
IndentWidth: 2
TabWidth: 2
ContinuationIndentWidth: 4
MaxEmptyLinesToKeep: 2
AccessModifierOffset: -2 # The private/protected/public has no indent in class
Standard: Cpp11
AllowAllParametersOfDeclarationOnNextLine: true
BinPackParameters: false
BinPackArguments: false
...
*.DS_Store
*.vs
*.user
*.pyc
*.vscode
*.idea
__pycache__/
_build/
build/
log/
.ipynb_checkpoints/
nohup.out
repos:
- repo: https://github.com/PaddlePaddle/mirrors-yapf.git
rev: 0d79c0c469bab64f7229c9aca2b1186ef47f0e37
hooks:
- id: yapf
files: \.py$
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: a11d9314b22d8f8c7556443875b731ef05965464
hooks:
- id: check-merge-conflict
- id: check-symlinks
- id: detect-private-key
files: (?!.*paddle)^.*$
- id: end-of-file-fixer
files: \.md$
- id: trailing-whitespace
files: \.md$
- repo: https://github.com/Lucas-C/pre-commit-hooks
rev: v1.0.1
hooks:
- id: forbid-crlf
files: \.md$
- id: remove-crlf
files: \.md$
- id: forbid-tabs
files: \.md$
- id: remove-tabs
files: \.md$
- repo: local
hooks:
- id: clang-format
name: clang-format
description: Format files with ClangFormat
entry: bash .clang_format.hook -i
language: system
files: \.(c|cc|cxx|cpp|cu|h|hpp|hxx|cuh|proto)$
Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserve.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
# repo template
# 模型名称
模型的具体名称,如:# ResNet50
目录
```
1. 简介
2. 复现精度
3. 数据集
4. 环境依赖
5. 快速开始
6. 代码结构与详细说明
7. 模型信息
```
**注意:** 目录可以使用[gh-md-toc](https://github.com/ekalinin/github-markdown-toc)生成
## 1. 简介
简单的介绍模型,以及模型的主要架构或主要功能,如果能给出效果图,可以在简介的下方直接贴上图片,展示模型效果。然后另起一行,按如下格式给出论文、参考代码及对应的链接。
**论文:** [title](url)
**参考repo:** [repo name](url)
## 2. 复现精度
给出论文中精度、参考代码的精度、本repo复现的精度、模型下载链接、模型大小,以表格的形式给出。如果超参数有差别,可以在表格中新增一列备注一下。
如果涉及到`轻量化骨干网络`,需要新增一列骨干网络的信息。
## 3. 数据集
给出数据集的链接,然后按格式描述数据集大小与数据集格式即可。
格式如下:
- 数据集大小:关于数据集大小的描述,如类别,数量,图像大小等等;
- 数据格式:关于数据集格式的说明
## 4. 环境依赖
主要分为两部分介绍,一部分是支持的硬件,另一部分是框架等环境的要求,格式如下:
- 硬件:
- 框架:
- PaddlePaddle >= 2.1.0
## 5. 快速开始
需要给出快速训练、预测、使用预训练模型预测、模型导出、模型基于inference模型推理的使用说明,同时基于demo图像,给出预测结果和推理结果,并将结果打印或者可视化出来。
## 6. 代码结构与详细说明
### 6.1 代码结构
需要用一小节描述整个项目的代码结构,用一小节描述项目的参数说明,之后各个小节详细的描述每个功能的使用说明。
### 6.2 参数说明
### 6.3 基础使用
配合部分重要配置参数,介绍模型训练、评估、预测、导出等过程。
### 6.4 模型部署
给出当前支持的推理部署方式以及相应的参考文档链接。
## 7. 模型信息
以表格的信息,给出模型和作者相关的信息。
| 信息 | 说明 |
| --- | --- |
| 发布者 | author |
| 时间 | 2021.11 |
| 框架版本 | Paddle 2.2 |
| 应用场景 | 图像分类 |
| 支持硬件 | GPU、CPU |
| 模型下载链接 | [trained model](url)[inference model](url) |
| 在线运行 | [notebook](url)[scripts](url) |
Markdown is supported
0% .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑!
想要评论请 注册