Skip to content

B
book

PaddlePaddle / book

通知 16
Star 4
Fork 0
B
book
提交 3a5ab5d9 编写于 作者: H helinwang 提交者： GitHub
浏览文件
操作

Merge pull request #160 from wangkuiyi/auto_convert_html_and_ipynb_in_pre_commit 

Auto convert html and ipynb in pre commit
上级 5ba755bb 37d41cfa
展开全部 显示空白变更内容
内联 并排
Showing with 974 addition and 995 deletion
.gitignore
浏览文件 @ 3a5ab5d9
deprecated
*~
pandoc.template pandoc.template
.DS_Store .DS_Store
\ No newline at end of file
.pre-commit-config.yaml
浏览文件 @ 3a5ab5d9
...@@ -10,11 +10,25 @@ ...@@ -10,11 +10,25 @@
- id: check-symlinks - id: check-symlinks
- id: detect-private-key - id: detect-private-key
- id: end-of-file-fixer - id: end-of-file-fixer
files: \.md$
- id: trailing-whitespace - id: trailing-whitespace
files: \.md$
- repo: git://github.com/Lucas-C/pre-commit-hooks - repo: git://github.com/Lucas-C/pre-commit-hooks
sha: v1.0.1 sha: v1.0.1
hooks: hooks:
- id: forbid-crlf - id: forbid-crlf
files: \.md$
- id: remove-crlf - id: remove-crlf
files: \.md$
- id: forbid-tabs - id: forbid-tabs
files: \.md$
- id: remove-tabs - id: remove-tabs
files: \.md$
- repo: local
hooks:
- id: convert-markdown-into-html
name: convert-markdown-into-html
description: "Convert README.md into index.html and README.en.md into index.en.html"
entry: python pre-commit-hooks/convert_markdown_into_html.py
language: system
files: \.md$
build.sh 已删除 100755 → 0
浏览文件 @ 5ba755bb
#!/bin/bash
for i in $(du -a | grep '\.\/.\+\/README.md' | cut -f 2); do
.tmpl/convert-markdown-into-html.sh $i > $(dirname $i)/index.html
done
for i in $(du -a | grep '\.\/.\+\/README.en.md' | cut -f 2); do
.tmpl/convert-markdown-into-html.sh $i > $(dirname $i)/index.en.html
done
fit_a_line/README.en.md
浏览文件 @ 3a5ab5d9
...@@ -202,4 +202,4 @@ This chapter introduces *Linear Regression* and how to train and test this model ...@@ -202,4 +202,4 @@ This chapter introduces *Linear Regression* and how to train and test this model
4. Bishop C M. Pattern recognition[J]. Machine Learning, 2006, 128. 4. Bishop C M. Pattern recognition[J]. Machine Learning, 2006, 128.
<br/> <br/>
<a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/"><img alt="知识共享许可协议" style="border-width:0" src="https://i.creativecommons.org/l/by-nc-sa/4.0/88x31.png" /></a><br /><span xmlns:dct="http://purl.org/dc/terms/" href="http://purl.org/dc/dcmitype/Text" property="dct:title" rel="dct:type">本教程</span> 由 <a xmlns:cc="http://creativecommons.org/ns#" href="http://book.paddlepaddle.org" property="cc:attributionName" rel="cc:attributionURL">PaddlePaddle</a> 创作，采用 <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/">知识共享 署名-非商业性使用-相同方式共享 4.0 国际 许可协议</a>进行许可。 <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/"><img alt="Common Creative License" style="border-width:0" src="https://i.creativecommons.org/l/by-nc-sa/4.0/88x31.png" /></a> This tutorial was created and published with [Creative Common License 4.0](http://creativecommons.org/licenses/by-nc-sa/4.0/).
fit_a_line/index.en.html
浏览文件 @ 3a5ab5d9
此差异已折叠。
fit_a_line/index.html
浏览文件 @ 3a5ab5d9
<html> <html>
<head> <head>
<script type="text/x-mathjax-config"> <script type="text/x-mathjax-config">
...@@ -5,8 +6,8 @@ ...@@ -5,8 +6,8 @@
extensions: ["tex2jax.js", "TeX/AMSsymbols.js", "TeX/AMSmath.js"], extensions: ["tex2jax.js", "TeX/AMSsymbols.js", "TeX/AMSmath.js"],
jax: ["input/TeX", "output/HTML-CSS"], jax: ["input/TeX", "output/HTML-CSS"],
tex2jax: { tex2jax: {
inlineMath: [ ['$','$'], ["\\(","\\)"] ], inlineMath: [ ['$','$'] ],
displayMath: [ ['$$','$$'], ["\\[","\\]"] ], displayMath: [ ['$$','$$'] ],
processEscapes: true processEscapes: true
}, },
"HTML-CSS": { availableFonts: ["TeX"] } "HTML-CSS": { availableFonts: ["TeX"] }
...@@ -86,14 +87,25 @@ $$MSE=\frac{1}{n}\sum_{i=1}^{n}{(\hat{Y_i}-Y_i)}^2$$ ...@@ -86,14 +87,25 @@ $$MSE=\frac{1}{n}\sum_{i=1}^{n}{(\hat{Y_i}-Y_i)}^2$$
3. 根据损失函数进行反向误差传播 （[backpropagation](https://en.wikipedia.org/wiki/Backpropagation)），将网络误差从输出层依次向前传递, 并更新网络中的参数。 3. 根据损失函数进行反向误差传播 （[backpropagation](https://en.wikipedia.org/wiki/Backpropagation)），将网络误差从输出层依次向前传递, 并更新网络中的参数。
4. 重复2~3步骤，直至网络训练误差达到规定的程度或训练轮次达到设定值。 4. 重复2~3步骤，直至网络训练误差达到规定的程度或训练轮次达到设定值。
## 数据集
### 数据集接口的封装
首先加载需要的包
## 数据准备 ```python
执行以下命令来准备数据: import paddle.v2 as paddle
```bash import paddle.v2.dataset.uci_housing as uci_housing
cd data && python prepare_data.py
``` ```
这段代码将从[UCI Housing Data Set](https://archive.ics.uci.edu/ml/datasets/Housing)下载数据并进行[预处理](#数据预处理)，最后数据将被分为训练集和测试集。
我们通过uci_housing模块引入了数据集合[UCI Housing Data Set](https://archive.ics.uci.edu/ml/datasets/Housing)
其中，在uci_housing模块中封装了：
1. 数据下载的过程。下载数据保存在~/.cache/paddle/dataset/uci_housing/housing.data。
2. [数据预处理](#数据预处理)的过程。
### 数据集介绍
这份数据集共506行，每行包含了波士顿郊区的一类房屋的相关信息及该类房屋价格的中位数。其各维属性的意义如下： 这份数据集共506行，每行包含了波士顿郊区的一类房屋的相关信息及该类房屋价格的中位数。其各维属性的意义如下：
| 属性名 | 解释 | 类型 | | 属性名 | 解释 | 类型 |
...@@ -131,89 +143,89 @@ cd data && python prepare_data.py ...@@ -131,89 +143,89 @@ cd data && python prepare_data.py
</p> </p>
#### 整理训练集与测试集 #### 整理训练集与测试集
我们将数据集分割为两份：一份用于调整模型的参数，即进行模型的训练，模型在这份数据集上的误差被称为**训练误差**；另外一份被用来测试，模型在这份数据集上的误差被称为**测试误差**。我们训练模型的目的是为了通过从训练数据中找到规律来预测未知的新数据，所以测试误差是更能反映模型表现的指标。分割数据的比例要考虑到两个因素：更多的训练数据会降低参数估计的方差，从而得到更可信的模型；而更多的测试数据会降低测试误差的方差，从而得到更可信的测试误差。一种常见的分割比例为$8:2$，感兴趣的读者朋友们也可以尝试不同的设置来观察这两种误差的变化。 我们将数据集分割为两份：一份用于调整模型的参数，即进行模型的训练，模型在这份数据集上的误差被称为**训练误差**；另外一份被用来测试，模型在这份数据集上的误差被称为**测试误差**。我们训练模型的目的是为了通过从训练数据中找到规律来预测未知的新数据，所以测试误差是更能反映模型表现的指标。分割数据的比例要考虑到两个因素：更多的训练数据会降低参数估计的方差，从而得到更可信的模型；而更多的测试数据会降低测试误差的方差，从而得到更可信的测试误差。我们这个例子中设置的分割比例为$8:2$
执行如下命令可以分割数据集，并将训练集和测试集的地址分别写入train.list 和 test.list两个文件中，供PaddlePaddle读取。
```python
python prepare_data.py -r 0.8 #默认使用8:2的比例进行分割
```
在更复杂的模型训练过程中，我们往往还会多使用一种数据集：验证集。因为复杂的模型中常常还有一些超参数（[Hyperparameter](https://en.wikipedia.org/wiki/Hyperparameter_optimization)）需要调节，所以我们会尝试多种超参数的组合来分别训练多个模型，然后对比它们在验证集上的表现选择相对最好的一组超参数，最后才使用这组参数下训练的模型在测试集上评估测试误差。由于本章训练的模型比较简单，我们暂且忽略掉这个过程。 在更复杂的模型训练过程中，我们往往还会多使用一种数据集：验证集。因为复杂的模型中常常还有一些超参数（[Hyperparameter](https://en.wikipedia.org/wiki/Hyperparameter_optimization)）需要调节，所以我们会尝试多种超参数的组合来分别训练多个模型，然后对比它们在验证集上的表现选择相对最好的一组超参数，最后才使用这组参数下训练的模型在测试集上评估测试误差。由于本章训练的模型比较简单，我们暂且忽略掉这个过程。
### 提供数据给PaddlePaddle ## 训练
准备好数据之后，我们使用一个Python data provider来为PaddlePaddle的训练过程提供数据。一个 data provider 就是一个Python函数，它会被PaddlePaddle的训练过程调用。在这个例子里，只需要读取已经保存好的数据，然后一行一行地返回给PaddlePaddle的训练进程即可。
```python
from paddle.trainer.PyDataProvider2 import *
import numpy as np
#定义数据的类型和维度
@provider(input_types=[dense_vector(13), dense_vector(1)])
def process(settings, input_file):
data = np.load(input_file.strip())
for row in data:
yield row[:-1].tolist(), row[-1:].tolist()
``` `fit_a_line/trainer.py`演示了训练的整体过程。
## 模型配置说明 ### 初始化PaddlePaddle
### 数据定义
首先，通过 `define_py_data_sources2` 来配置PaddlePaddle从上面的`dataprovider.py`里读入训练数据和测试数据。 PaddlePaddle接受从命令行读入的配置信息，例如这里我们传入一个名为`is_predict`的变量来控制模型在训练和测试时的不同结构。
```python ```python
from paddle.trainer_config_helpers import * paddle.init(use_gpu=False, trainer_count=1)
```
is_predict = get_config_arg('is_predict', bool, False) ### 模型配置
define_py_data_sources2( 线性回归的模型其实就是一个采用线性激活函数（linear activation，`LinearActivation`）的全连接层（fully-connected layer，`fc_layer`）：
train_list='data/train.list',
test_list='data/test.list',
module='dataprovider',
obj='process')
```python
x = paddle.layer.data(name='x', type=paddle.data_type.dense_vector(13))
y_predict = paddle.layer.fc(input=x,
size=1,
act=paddle.activation.Linear())
y = paddle.layer.data(name='y', type=paddle.data_type.dense_vector(1))
cost = paddle.layer.regression_cost(input=y_predict, label=y)
``` ```
### 创建参数
### 算法配置
接着，指定模型优化算法的细节。由于线性回归模型比较简单，我们只要设置基本的`batch_size`即可，它指定每次更新参数的时候使用多少条数据计算梯度信息。
```python ```python
settings(batch_size=2) parameters = paddle.parameters.create(cost)
``` ```
### 网络结构 ### 创建Trainer
最后，使用`fc_layer`和`LinearActivation`来表示线性回归的模型本身。
```python ```python
#输入数据，13维的房屋信息 optimizer = paddle.optimizer.Momentum(momentum=0)
x = data_layer(name='x', size=13)
y_predict = fc_layer( trainer = paddle.trainer.SGD(cost=cost,
input=x, parameters=parameters,
param_attr=ParamAttr(name='w'), update_equation=optimizer)
size=1,
act=LinearActivation(),
bias_attr=ParamAttr(name='b'))
if not is_predict: #训练时，我们使用MSE，即regression_cost作为损失函数
y = data_layer(name='y', size=1)
cost = regression_cost(input=y_predict, label=y)
outputs(cost) #训练时输出MSE来监控损失的变化
else: #测试时，输出预测值
outputs(y_predict)
``` ```
## 训练模型 ### 读取数据且打印训练的中间信息
在对应代码的根目录下执行PaddlePaddle的命令行训练程序。这里指定模型配置文件为`trainer_config.py`，训练30轮，结果保存在`output`路径下。
```bash PaddlePaddle提供一个
./train.sh [reader机制](https://github.com/PaddlePaddle/Paddle/tree/develop/doc/design/reader)
来读取数据。 Reader返回的数据可以包括多列，我们需要一个Python dict把列
序号映射到网络里的数据层。
```python
feeding={'x': 0, 'y': 1}
``` ```
## 应用模型 此外，我们还可以提供一个 event handler，来打印训练的进度：
现在来看下如何使用已经训练好的模型进行预测。
```bash ```python
python predict.py # event_handler to print training and testing info
def event_handler(event):
if isinstance(event, paddle.event.EndIteration):
if event.batch_id % 100 == 0:
print "Pass %d, Batch %d, Cost %f" % (
event.pass_id, event.batch_id, event.cost)
if isinstance(event, paddle.event.EndPass):
result = trainer.test(
reader=paddle.batch(
uci_housing.test(), batch_size=2),
feeding=feeding)
print "Test %d, Cost %f" % (event.pass_id, result.cost)
``` ```
这里默认使用`output/pass-00029`中保存的模型进行预测，并将数据中的房价与预测结果进行对比，结果保存在 `predictions.png`中。
如果你想使用别的模型或者其它的数据进行预测，只要传入新的路径即可： ### 开始训练
```bash
python predict.py -m output/pass-00020 -t data/housing.test.npy ```python
trainer.train(
reader=paddle.batch(
paddle.reader.shuffle(
uci_housing.train(), buf_size=500),
batch_size=2),
feeding=feeding,
event_handler=event_handler,
num_passes=30)
``` ```
## 总结 ## 总结
...@@ -228,6 +240,7 @@ python predict.py -m output/pass-00020 -t data/housing.test.npy ...@@ -228,6 +240,7 @@ python predict.py -m output/pass-00020 -t data/housing.test.npy
<br/> <br/>
<a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/"><img alt="知识共享许可协议" style="border-width:0" src="https://i.creativecommons.org/l/by-nc-sa/4.0/88x31.png" /></a><br /><span xmlns:dct="http://purl.org/dc/terms/" href="http://purl.org/dc/dcmitype/Text" property="dct:title" rel="dct:type">本教程</span> 由 <a xmlns:cc="http://creativecommons.org/ns#" href="http://book.paddlepaddle.org" property="cc:attributionName" rel="cc:attributionURL">PaddlePaddle</a> 创作，采用 <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/">知识共享 署名-非商业性使用-相同方式共享 4.0 国际 许可协议</a>进行许可。 <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/"><img alt="知识共享许可协议" style="border-width:0" src="https://i.creativecommons.org/l/by-nc-sa/4.0/88x31.png" /></a><br /><span xmlns:dct="http://purl.org/dc/terms/" href="http://purl.org/dc/dcmitype/Text" property="dct:title" rel="dct:type">本教程</span> 由 <a xmlns:cc="http://creativecommons.org/ns#" href="http://book.paddlepaddle.org" property="cc:attributionName" rel="cc:attributionURL">PaddlePaddle</a> 创作，采用 <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/">知识共享 署名-非商业性使用-相同方式共享 4.0 国际 许可协议</a>进行许可。
</div> </div>
<!-- You can change the lines below now. --> <!-- You can change the lines below now. -->
......
gan/index.html
浏览文件 @ 3a5ab5d9
<html> <html>
<head> <head>
<script type="text/x-mathjax-config"> <script type="text/x-mathjax-config">
...@@ -5,8 +6,8 @@ ...@@ -5,8 +6,8 @@
extensions: ["tex2jax.js", "TeX/AMSsymbols.js", "TeX/AMSmath.js"], extensions: ["tex2jax.js", "TeX/AMSsymbols.js", "TeX/AMSmath.js"],
jax: ["input/TeX", "output/HTML-CSS"], jax: ["input/TeX", "output/HTML-CSS"],
tex2jax: { tex2jax: {
inlineMath: [ ['$','$'], ["\\(","\\)"] ], inlineMath: [ ['$','$'] ],
displayMath: [ ['$$','$$'], ["\\[","\\]"] ], displayMath: [ ['$$','$$'] ],
processEscapes: true processEscapes: true
}, },
"HTML-CSS": { availableFonts: ["TeX"] } "HTML-CSS": { availableFonts: ["TeX"] }
...@@ -40,6 +41,7 @@ ...@@ -40,6 +41,7 @@
<!-- This block will be replaced by each markdown file content. Please do not change lines below.--> <!-- This block will be replaced by each markdown file content. Please do not change lines below.-->
<div id="markdown" style='display:none'> <div id="markdown" style='display:none'>
TODO: Write about https://github.com/PaddlePaddle/Paddle/tree/develop/demo/gan TODO: Write about https://github.com/PaddlePaddle/Paddle/tree/develop/demo/gan
</div> </div>
<!-- You can change the lines below now. --> <!-- You can change the lines below now. -->
......
image_caption/index.html
浏览文件 @ 3a5ab5d9
<html> <html>
<head> <head>
<script type="text/x-mathjax-config"> <script type="text/x-mathjax-config">
...@@ -5,8 +6,8 @@ ...@@ -5,8 +6,8 @@
extensions: ["tex2jax.js", "TeX/AMSsymbols.js", "TeX/AMSmath.js"], extensions: ["tex2jax.js", "TeX/AMSsymbols.js", "TeX/AMSmath.js"],
jax: ["input/TeX", "output/HTML-CSS"], jax: ["input/TeX", "output/HTML-CSS"],
tex2jax: { tex2jax: {
inlineMath: [ ['$','$'], ["\\(","\\)"] ], inlineMath: [ ['$','$'] ],
displayMath: [ ['$$','$$'], ["\\[","\\]"] ], displayMath: [ ['$$','$$'] ],
processEscapes: true processEscapes: true
}, },
"HTML-CSS": { availableFonts: ["TeX"] } "HTML-CSS": { availableFonts: ["TeX"] }
...@@ -39,6 +40,7 @@ ...@@ -39,6 +40,7 @@
<!-- This block will be replaced by each markdown file content. Please do not change lines below.--> <!-- This block will be replaced by each markdown file content. Please do not change lines below.-->
<div id="markdown" style='display:none'> <div id="markdown" style='display:none'>
</div> </div>
<!-- You can change the lines below now. --> <!-- You can change the lines below now. -->
......
image_classification/index.en.html
浏览文件 @ 3a5ab5d9
<html> <html>
<head> <head>
<script type="text/x-mathjax-config"> <script type="text/x-mathjax-config">
...@@ -5,8 +6,8 @@ ...@@ -5,8 +6,8 @@
extensions: ["tex2jax.js", "TeX/AMSsymbols.js", "TeX/AMSmath.js"], extensions: ["tex2jax.js", "TeX/AMSsymbols.js", "TeX/AMSmath.js"],
jax: ["input/TeX", "output/HTML-CSS"], jax: ["input/TeX", "output/HTML-CSS"],
tex2jax: { tex2jax: {
inlineMath: [ ['$','$'], ["\\(","\\)"] ], inlineMath: [ ['$','$'] ],
displayMath: [ ['$$','$$'], ["\\[","\\]"] ], displayMath: [ ['$$','$$'] ],
processEscapes: true processEscapes: true
}, },
"HTML-CSS": { availableFonts: ["TeX"] } "HTML-CSS": { availableFonts: ["TeX"] }
...@@ -589,6 +590,7 @@ Traditional image classification methods involve multiple stages of processing a ...@@ -589,6 +590,7 @@ Traditional image classification methods involve multiple stages of processing a
<br/> <br/>
<a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/"><img alt="知识共享许可协议" style="border-width:0" src="https://i.creativecommons.org/l/by-nc-sa/4.0/88x31.png" /></a><br /><span xmlns:dct="http://purl.org/dc/terms/" href="http://purl.org/dc/dcmitype/Text" property="dct:title" rel="dct:type">本教程</span> 由 <a xmlns:cc="http://creativecommons.org/ns#" href="http://book.paddlepaddle.org" property="cc:attributionName" rel="cc:attributionURL">PaddlePaddle</a> 创作，采用 <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/">知识共享 署名-非商业性使用-相同方式共享 4.0 国际 许可协议</a>进行许可。 <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/"><img alt="知识共享许可协议" style="border-width:0" src="https://i.creativecommons.org/l/by-nc-sa/4.0/88x31.png" /></a><br /><span xmlns:dct="http://purl.org/dc/terms/" href="http://purl.org/dc/dcmitype/Text" property="dct:title" rel="dct:type">本教程</span> 由 <a xmlns:cc="http://creativecommons.org/ns#" href="http://book.paddlepaddle.org" property="cc:attributionName" rel="cc:attributionURL">PaddlePaddle</a> 创作，采用 <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/">知识共享 署名-非商业性使用-相同方式共享 4.0 国际 许可协议</a>进行许可。
</div> </div>
<!-- You can change the lines below now. --> <!-- You can change the lines below now. -->
......
image_classification/index.html
浏览文件 @ 3a5ab5d9
<html> <html>
<head> <head>
<script type="text/x-mathjax-config"> <script type="text/x-mathjax-config">
...@@ -5,8 +6,8 @@ ...@@ -5,8 +6,8 @@
extensions: ["tex2jax.js", "TeX/AMSsymbols.js", "TeX/AMSmath.js"], extensions: ["tex2jax.js", "TeX/AMSsymbols.js", "TeX/AMSmath.js"],
jax: ["input/TeX", "output/HTML-CSS"], jax: ["input/TeX", "output/HTML-CSS"],
tex2jax: { tex2jax: {
inlineMath: [ ['$','$'], ["\\(","\\)"] ], inlineMath: [ ['$','$'] ],
displayMath: [ ['$$','$$'], ["\\[","\\]"] ], displayMath: [ ['$$','$$'] ],
processEscapes: true processEscapes: true
}, },
"HTML-CSS": { availableFonts: ["TeX"] } "HTML-CSS": { availableFonts: ["TeX"] }
...@@ -535,6 +536,7 @@ Test with Pass 0, {'classification_error_evaluator': 0.885200023651123} ...@@ -535,6 +536,7 @@ Test with Pass 0, {'classification_error_evaluator': 0.885200023651123}
<br/> <br/>
<a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/"><img alt="知识共享许可协议" style="border-width:0" src="https://i.creativecommons.org/l/by-nc-sa/4.0/88x31.png" /></a><br /><span xmlns:dct="http://purl.org/dc/terms/" href="http://purl.org/dc/dcmitype/Text" property="dct:title" rel="dct:type">本教程</span> 由 <a xmlns:cc="http://creativecommons.org/ns#" href="http://book.paddlepaddle.org" property="cc:attributionName" rel="cc:attributionURL">PaddlePaddle</a> 创作，采用 <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/">知识共享 署名-非商业性使用-相同方式共享 4.0 国际 许可协议</a>进行许可。 <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/"><img alt="知识共享许可协议" style="border-width:0" src="https://i.creativecommons.org/l/by-nc-sa/4.0/88x31.png" /></a><br /><span xmlns:dct="http://purl.org/dc/terms/" href="http://purl.org/dc/dcmitype/Text" property="dct:title" rel="dct:type">本教程</span> 由 <a xmlns:cc="http://creativecommons.org/ns#" href="http://book.paddlepaddle.org" property="cc:attributionName" rel="cc:attributionURL">PaddlePaddle</a> 创作，采用 <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/">知识共享 署名-非商业性使用-相同方式共享 4.0 国际 许可协议</a>进行许可。
</div> </div>
<!-- You can change the lines below now. --> <!-- You can change the lines below now. -->
......
image_detection/index.html
浏览文件 @ 3a5ab5d9
<html> <html>
<head> <head>
<script type="text/x-mathjax-config"> <script type="text/x-mathjax-config">
...@@ -5,8 +6,8 @@ ...@@ -5,8 +6,8 @@
extensions: ["tex2jax.js", "TeX/AMSsymbols.js", "TeX/AMSmath.js"], extensions: ["tex2jax.js", "TeX/AMSsymbols.js", "TeX/AMSmath.js"],
jax: ["input/TeX", "output/HTML-CSS"], jax: ["input/TeX", "output/HTML-CSS"],
tex2jax: { tex2jax: {
inlineMath: [ ['$','$'], ["\\(","\\)"] ], inlineMath: [ ['$','$'] ],
displayMath: [ ['$$','$$'], ["\\[","\\]"] ], displayMath: [ ['$$','$$'] ],
processEscapes: true processEscapes: true
}, },
"HTML-CSS": { availableFonts: ["TeX"] } "HTML-CSS": { availableFonts: ["TeX"] }
...@@ -39,6 +40,7 @@ ...@@ -39,6 +40,7 @@
<!-- This block will be replaced by each markdown file content. Please do not change lines below.--> <!-- This block will be replaced by each markdown file content. Please do not change lines below.-->
<div id="markdown" style='display:none'> <div id="markdown" style='display:none'>
</div> </div>
<!-- You can change the lines below now. --> <!-- You can change the lines below now. -->
......
image_qa/index.html
浏览文件 @ 3a5ab5d9
<html> <html>
<head> <head>
<script type="text/x-mathjax-config"> <script type="text/x-mathjax-config">
...@@ -5,8 +6,8 @@ ...@@ -5,8 +6,8 @@
extensions: ["tex2jax.js", "TeX/AMSsymbols.js", "TeX/AMSmath.js"], extensions: ["tex2jax.js", "TeX/AMSsymbols.js", "TeX/AMSmath.js"],
jax: ["input/TeX", "output/HTML-CSS"], jax: ["input/TeX", "output/HTML-CSS"],
tex2jax: { tex2jax: {
inlineMath: [ ['$','$'], ["\\(","\\)"] ], inlineMath: [ ['$','$'] ],
displayMath: [ ['$$','$$'], ["\\[","\\]"] ], displayMath: [ ['$$','$$'] ],
processEscapes: true processEscapes: true
}, },
"HTML-CSS": { availableFonts: ["TeX"] } "HTML-CSS": { availableFonts: ["TeX"] }
...@@ -39,6 +40,7 @@ ...@@ -39,6 +40,7 @@
<!-- This block will be replaced by each markdown file content. Please do not change lines below.--> <!-- This block will be replaced by each markdown file content. Please do not change lines below.-->
<div id="markdown" style='display:none'> <div id="markdown" style='display:none'>
</div> </div>
<!-- You can change the lines below now. --> <!-- You can change the lines below now. -->
......
index.html
浏览文件 @ 3a5ab5d9
<html> <html>
<head> <head>
<meta http-equiv="refresh" content="0; url=https://github.com/paddlepaddle/book" /> <script type="text/x-mathjax-config">
MathJax.Hub.Config({
extensions: ["tex2jax.js", "TeX/AMSsymbols.js", "TeX/AMSmath.js"],
jax: ["input/TeX", "output/HTML-CSS"],
tex2jax: {
inlineMath: [ ['$','$'] ],
displayMath: [ ['$$','$$'] ],
processEscapes: true
},
"HTML-CSS": { availableFonts: ["TeX"] }
});
</script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js" async></script>
<script type="text/javascript" src="../.tmpl/marked.js">
</script>
<link href="http://cdn.bootcss.com/highlight.js/9.9.0/styles/darcula.min.css" rel="stylesheet">
<script src="http://cdn.bootcss.com/highlight.js/9.9.0/highlight.min.js"></script>
<link href="http://cdn.bootcss.com/bootstrap/4.0.0-alpha.6/css/bootstrap.min.css" rel="stylesheet">
<link href="https://cdn.jsdelivr.net/perfect-scrollbar/0.6.14/css/perfect-scrollbar.min.css" rel="stylesheet">
<link href="../.tmpl/github-markdown.css" rel='stylesheet'>
</head> </head>
<style type="text/css" >
.markdown-body {
box-sizing: border-box;
min-width: 200px;
max-width: 980px;
margin: 0 auto;
padding: 45px;
}
</style>
<body> <body>
<a href="https://github.com/paddlepaddle/book">Please access github home page</a>
<div id="context" class="container markdown-body">
</div>
<!-- This block will be replaced by each markdown file content. Please do not change lines below.-->
<div id="markdown" style='display:none'>
# 深度学习入门
1. 新手入门 [[fit_a_line](fit_a_line/)] [[html](http://book.paddlepaddle.org/fit_a_line)]
1. 识别数字 [[recognize_digits](recognize_digits/)] [[html](http://book.paddlepaddle.org/recognize_digits)]
1. 图像分类 [[image_classification](image_classification/)] [[html](http://book.paddlepaddle.org/image_classification)]
1. 词向量 [[word2vec](word2vec/)] [[html](http://book.paddlepaddle.org/word2vec)]
1. 情感分析 [[understand_sentiment](understand_sentiment/)] [[html](http://book.paddlepaddle.org/understand_sentiment)]
1. 语义角色标注 [[label_semantic_roles](label_semantic_roles/)] [[html](http://book.paddlepaddle.org/label_semantic_roles)]
1. 机器翻译 [[machine_translation](machine_translation/)] [[html](http://book.paddlepaddle.org/machine_translation)]
1. 个性化推荐 [[recommender_system](recommender_system/)] [[html](http://book.paddlepaddle.org/recommender_system)]
<br/>
<a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/"><img alt="知识共享许可协议" style="border-width:0" src="https://i.creativecommons.org/l/by-nc-sa/4.0/88x31.png" /></a><br /><span xmlns:dct="http://purl.org/dc/terms/" href="http://purl.org/dc/dcmitype/Text" property="dct:title" rel="dct:type">本教程</span> 由 <a xmlns:cc="http://creativecommons.org/ns#" href="http://book.paddlepaddle.org" property="cc:attributionName" rel="cc:attributionURL">PaddlePaddle</a> 创作，采用 <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/">知识共享 署名-非商业性使用-相同方式共享 4.0 国际 许可协议</a>进行许可。
</div>
<!-- You can change the lines below now. -->
<script type="text/javascript">
marked.setOptions({
renderer: new marked.Renderer(),
gfm: true,
breaks: false,
smartypants: true,
highlight: function(code, lang) {
code = code.replace(/&amp;/g, "&")
code = code.replace(/&gt;/g, ">")
code = code.replace(/&lt;/g, "<")
code = code.replace(/&nbsp;/g, " ")
return hljs.highlightAuto(code, [lang]).value;
}
});
document.getElementById("context").innerHTML = marked(
document.getElementById("markdown").innerHTML)
</script>
</body> </body>
label_semantic_roles/index.en.html
浏览文件 @ 3a5ab5d9
<html> <html>
<head> <head>
<script type="text/x-mathjax-config"> <script type="text/x-mathjax-config">
...@@ -5,8 +6,8 @@ ...@@ -5,8 +6,8 @@
extensions: ["tex2jax.js", "TeX/AMSsymbols.js", "TeX/AMSmath.js"], extensions: ["tex2jax.js", "TeX/AMSsymbols.js", "TeX/AMSmath.js"],
jax: ["input/TeX", "output/HTML-CSS"], jax: ["input/TeX", "output/HTML-CSS"],
tex2jax: { tex2jax: {
inlineMath: [ ['$','$'], ["\\(","\\)"] ], inlineMath: [ ['$','$'] ],
displayMath: [ ['$$','$$'], ["\\[","\\]"] ], displayMath: [ ['$$','$$'] ],
processEscapes: true processEscapes: true
}, },
"HTML-CSS": { availableFonts: ["TeX"] } "HTML-CSS": { availableFonts: ["TeX"] }
...@@ -63,34 +64,20 @@ Standard SRL system mostly builds on top of Syntactic Analysis and contains five ...@@ -63,34 +64,20 @@ Standard SRL system mostly builds on top of Syntactic Analysis and contains five
<div align="center"> <div align="center">
<img src="image/dependency_parsing.png" width = "80%" align=center /><br> <img src="image/dependency_parsing_en.png" width = "80%" align=center /><br>
Fig 1. Syntactic parse tree Fig 1. Syntactic parse tree
</div> </div>
核心关系-> HED
定中关系-> ATT
主谓关系-> SBV
状中结构-> ADV
介宾关系-> POB
右附加关系-> RAD
动宾关系-> VOB
标点-> WP
However, complete syntactic analysis requires identifying the relation among all constitutes and the performance of SRL is sensitive to the precision of syntactic analysis, which makes SRL a very challenging task. To reduce the complexity and obtain some syntactic structure information, we often use shallow syntactic analysis. Shallow Syntactic Analysis is also called partial parsing or chunking. Unlike complete syntactic analysis which requires the construction of the complete parsing tree, Shallow Syntactic Analysis only need to identify some independent components with relatively simple structure, such as verb phrases (chunk). To avoid difficulties in constructing a syntactic tree with high accuracy, some work\[[1](#Reference)\] proposed semantic chunking based SRL methods, which convert SRL as a sequence tagging problem. Sequence tagging tasks classify syntactic chunks using BIO representation. For syntactic chunks forming a chunk of type A, the first chunk receives the B-A tag (Begin), the remaining ones receive the tag I-A (Inside), and all chunks outside receive the tag O-A. However, complete syntactic analysis requires identifying the relation among all constitutes and the performance of SRL is sensitive to the precision of syntactic analysis, which makes SRL a very challenging task. To reduce the complexity and obtain some syntactic structure information, we often use shallow syntactic analysis. Shallow Syntactic Analysis is also called partial parsing or chunking. Unlike complete syntactic analysis which requires the construction of the complete parsing tree, Shallow Syntactic Analysis only need to identify some independent components with relatively simple structure, such as verb phrases (chunk). To avoid difficulties in constructing a syntactic tree with high accuracy, some work\[[1](#Reference)\] proposed semantic chunking based SRL methods, which convert SRL as a sequence tagging problem. Sequence tagging tasks classify syntactic chunks using BIO representation. For syntactic chunks forming a chunk of type A, the first chunk receives the B-A tag (Begin), the remaining ones receive the tag I-A (Inside), and all chunks outside receive the tag O-A.
The BIO representation of above example is shown in Fig.1. The BIO representation of above example is shown in Fig.1.
<div align="center"> <div align="center">
<img src="image/bio_example.png" width = "90%" align=center /><br> <img src="image/bio_example_en.png" width = "90%" align=center /><br>
Fig 2. BIO represention Fig 2. BIO represention
</div> </div>
输入序列-> input sequence
语块-> chunk
标注序列-> label sequence
角色-> role
This example illustrates the simplicity of sequence tagging because (1) shallow syntactic analysis reduces the precision requirement of syntactic analysis; (2) pruning candidate arguments is removed; 3) argument identification and tagging are finished at the same time. Such unified methods simplify the procedure, reduce the risk of accumulating errors and boost the performance further. This example illustrates the simplicity of sequence tagging because (1) shallow syntactic analysis reduces the precision requirement of syntactic analysis; (2) pruning candidate arguments is removed; 3) argument identification and tagging are finished at the same time. Such unified methods simplify the procedure, reduce the risk of accumulating errors and boost the performance further.
In this tutorial, our SRL system is built as an end-to-end system via a neural network. We take only text sequences, without using any syntactic parsing results or complex hand-designed features. We give public dataset [CoNLL-2004 and CoNLL-2005 Shared Tasks](http://www.cs.upc.edu/~srlconll/) as an example to illustrate: given a sentence with predicates marked, identify the corresponding arguments and their semantic roles by sequence tagging method. In this tutorial, our SRL system is built as an end-to-end system via a neural network. We take only text sequences, without using any syntactic parsing results or complex hand-designed features. We give public dataset [CoNLL-2004 and CoNLL-2005 Shared Tasks](http://www.cs.upc.edu/~srlconll/) as an example to illustrate: given a sentence with predicates marked, identify the corresponding arguments and their semantic roles by sequence tagging method.
...@@ -112,13 +99,10 @@ The operation of a single LSTM cell contain 3 parts: (1) input-to-hidden: map in ...@@ -112,13 +99,10 @@ The operation of a single LSTM cell contain 3 parts: (1) input-to-hidden: map in
Fig.3 illustrate the final stacked recurrent neural networks. Fig.3 illustrate the final stacked recurrent neural networks.
<p align="center"> <p align="center">
<img src="./image/stacked_lstm.png" width = "40%" align=center><br> <img src="./image/stacked_lstm_en.png" width = "40%" align=center><br>
Fig 3. Stacked Recurrent Neural Networks Fig 3. Stacked Recurrent Neural Networks
</p> </p>
线性变换-> linear transformation
输入层到隐层-> input-to-hidden
### Bidirectional Recurrent Neural Network ### Bidirectional Recurrent Neural Network
LSTMs can summarize the history of previous inputs seen up to now, but can not see the future. In most of NLP (natural language processing) tasks, the entire sentences are ready to use. Therefore, sequential learning might be much efficient if the future can be encoded as well like histories. LSTMs can summarize the history of previous inputs seen up to now, but can not see the future. In most of NLP (natural language processing) tasks, the entire sentences are ready to use. Therefore, sequential learning might be much efficient if the future can be encoded as well like histories.
...@@ -127,15 +111,10 @@ To address the above drawbacks, we can design bidirectional recurrent neural net ...@@ -127,15 +111,10 @@ To address the above drawbacks, we can design bidirectional recurrent neural net
<p align="center"> <p align="center">
<img src="./image/bidirectional_stacked_lstm.png" width = "60%" align=center><br> <img src="./image/bidirectional_stacked_lstm_en.png" width = "60%" align=center><br>
Fig 4. Bidirectional LSTMs Fig 4. Bidirectional LSTMs
</p> </p>
线性变换-> linear transformation
输入层到隐层-> input-to-hidden
正向处理输出序列->process sequence in the forward direction
反向处理上一层序列-> process sequence from the previous layer in backward direction
Note that, this bidirectional RNNs is different with the one proposed by Bengio et al. in machine translation tasks \[[3](#Reference), [4](#Reference)\]. We will introduce another bidirectional RNNs in the following tasks[machine translation](https://github.com/PaddlePaddle/book/blob/develop/machine_translation/README.md) Note that, this bidirectional RNNs is different with the one proposed by Bengio et al. in machine translation tasks \[[3](#Reference), [4](#Reference)\]. We will introduce another bidirectional RNNs in the following tasks[machine translation](https://github.com/PaddlePaddle/book/blob/develop/machine_translation/README.md)
### Conditional Random Field ### Conditional Random Field
...@@ -197,18 +176,10 @@ After modification, the model is as follows: ...@@ -197,18 +176,10 @@ After modification, the model is as follows:
<div align="center"> <div align="center">
<img src="image/db_lstm_network.png" width = "60%" align=center /><br> <img src="image/db_lstm_en.png" width = "60%" align=center /><br>
Fig 6. DB-LSTM for SRL tasks Fig 6. DB-LSTM for SRL tasks
</div> </div>
论元-> argu
谓词-> pred
谓词上下文-> ctx-p
谓词上下文区域标记-> $m_r$
输入-> input
原句-> sentence
反向LSTM-> LSTM Reverse
## Data Preparation ## Data Preparation
In the tutorial, we use [CoNLL 2005](http://www.cs.upc.edu/~srlconll/) SRL task open dataset as an example. It is important to note that the training set and development set of the CoNLL 2005 SRL task are not free to download after the competition. Currently, only the test set can be obtained, including 23 sections of the Wall Street Journal and three sections of the Brown corpus. In this tutorial, we use the WSJ corpus as the training dataset to explain the model. However, since the training set is small, if you want to train a usable neural network SRL system, consider paying for the full corpus. In the tutorial, we use [CoNLL 2005](http://www.cs.upc.edu/~srlconll/) SRL task open dataset as an example. It is important to note that the training set and development set of the CoNLL 2005 SRL task are not free to download after the competition. Currently, only the test set can be obtained, including 23 sections of the Wall Street Journal and three sections of the Brown corpus. In this tutorial, we use the WSJ corpus as the training dataset to explain the model. However, since the training set is small, if you want to train a usable neural network SRL system, consider paying for the full corpus.
...@@ -542,6 +513,7 @@ Semantic Role Labeling is an important intermediate step in a wide range of natu ...@@ -542,6 +513,7 @@ Semantic Role Labeling is an important intermediate step in a wide range of natu
<br/> <br/>
<a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/"><img alt="知识共享许可协议" style="border-width:0" src="https://i.creativecommons.org/l/by-nc-sa/4.0/88x31.png" /></a><br /><span xmlns:dct="http://purl.org/dc/terms/" href="http://purl.org/dc/dcmitype/Text" property="dct:title" rel="dct:type">本教程</span> 由 <a xmlns:cc="http://creativecommons.org/ns#" href="http://book.paddlepaddle.org" property="cc:attributionName" rel="cc:attributionURL">PaddlePaddle</a> 创作，采用 <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/">知识共享 署名-非商业性使用-相同方式共享 4.0 国际 许可协议</a>进行许可。 <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/"><img alt="知识共享许可协议" style="border-width:0" src="https://i.creativecommons.org/l/by-nc-sa/4.0/88x31.png" /></a><br /><span xmlns:dct="http://purl.org/dc/terms/" href="http://purl.org/dc/dcmitype/Text" property="dct:title" rel="dct:type">本教程</span> 由 <a xmlns:cc="http://creativecommons.org/ns#" href="http://book.paddlepaddle.org" property="cc:attributionName" rel="cc:attributionURL">PaddlePaddle</a> 创作，采用 <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/">知识共享 署名-非商业性使用-相同方式共享 4.0 国际 许可协议</a>进行许可。
</div> </div>
<!-- You can change the lines below now. --> <!-- You can change the lines below now. -->
......
label_semantic_roles/index.html
浏览文件 @ 3a5ab5d9
<html> <html>
<head> <head>
<script type="text/x-mathjax-config"> <script type="text/x-mathjax-config">
...@@ -5,8 +6,8 @@ ...@@ -5,8 +6,8 @@
extensions: ["tex2jax.js", "TeX/AMSsymbols.js", "TeX/AMSmath.js"], extensions: ["tex2jax.js", "TeX/AMSsymbols.js", "TeX/AMSmath.js"],
jax: ["input/TeX", "output/HTML-CSS"], jax: ["input/TeX", "output/HTML-CSS"],
tex2jax: { tex2jax: {
inlineMath: [ ['$','$'], ["\\(","\\)"] ], inlineMath: [ ['$','$'] ],
displayMath: [ ['$$','$$'], ["\\[","\\]"] ], displayMath: [ ['$$','$$'] ],
processEscapes: true processEscapes: true
}, },
"HTML-CSS": { availableFonts: ["TeX"] } "HTML-CSS": { availableFonts: ["TeX"] }
...@@ -509,6 +510,7 @@ trainer.train( ...@@ -509,6 +510,7 @@ trainer.train(
<br/> <br/>
<a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/"><img alt="知识共享许可协议" style="border-width:0" src="https://i.creativecommons.org/l/by-nc-sa/4.0/88x31.png" /></a><br /><span xmlns:dct="http://purl.org/dc/terms/" href="http://purl.org/dc/dcmitype/Text" property="dct:title" rel="dct:type">本教程</span> 由 <a xmlns:cc="http://creativecommons.org/ns#" href="http://book.paddlepaddle.org" property="cc:attributionName" rel="cc:attributionURL">PaddlePaddle</a> 创作，采用 <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/">知识共享 署名-非商业性使用-相同方式共享 4.0 国际 许可协议</a>进行许可。 <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/"><img alt="知识共享许可协议" style="border-width:0" src="https://i.creativecommons.org/l/by-nc-sa/4.0/88x31.png" /></a><br /><span xmlns:dct="http://purl.org/dc/terms/" href="http://purl.org/dc/dcmitype/Text" property="dct:title" rel="dct:type">本教程</span> 由 <a xmlns:cc="http://creativecommons.org/ns#" href="http://book.paddlepaddle.org" property="cc:attributionName" rel="cc:attributionURL">PaddlePaddle</a> 创作，采用 <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/">知识共享 署名-非商业性使用-相同方式共享 4.0 国际 许可协议</a>进行许可。
</div> </div>
<!-- You can change the lines below now. --> <!-- You can change the lines below now. -->
......
machine_translation/index.en.html
浏览文件 @ 3a5ab5d9
<html> <html>
<head> <head>
<script type="text/x-mathjax-config"> <script type="text/x-mathjax-config">
...@@ -5,8 +6,8 @@ ...@@ -5,8 +6,8 @@
extensions: ["tex2jax.js", "TeX/AMSsymbols.js", "TeX/AMSmath.js"], extensions: ["tex2jax.js", "TeX/AMSsymbols.js", "TeX/AMSmath.js"],
jax: ["input/TeX", "output/HTML-CSS"], jax: ["input/TeX", "output/HTML-CSS"],
tex2jax: { tex2jax: {
inlineMath: [ ['$','$'], ["\\(","\\)"] ], inlineMath: [ ['$','$'] ],
displayMath: [ ['$$','$$'], ["\\[","\\]"] ], displayMath: [ ['$$','$$'] ],
processEscapes: true processEscapes: true
}, },
"HTML-CSS": { availableFonts: ["TeX"] } "HTML-CSS": { availableFonts: ["TeX"] }
...@@ -764,6 +765,7 @@ End-to-end neural machine translation is a recently developed way to perform mac ...@@ -764,6 +765,7 @@ End-to-end neural machine translation is a recently developed way to perform mac
<br/> <br/>
<a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/"><img alt="知识共享许可协议" style="border-width:0" src="https://i.creativecommons.org/l/by-nc-sa/4.0/88x31.png" /></a><br /><span xmlns:dct="http://purl.org/dc/terms/" href="http://purl.org/dc/dcmitype/Text" property="dct:title" rel="dct:type">本教程</span> 由 <a xmlns:cc="http://creativecommons.org/ns#" href="http://book.paddlepaddle.org" property="cc:attributionName" rel="cc:attributionURL">PaddlePaddle</a> 创作，采用 <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/">知识共享 署名-非商业性使用-相同方式共享 4.0 国际 许可协议</a>进行许可。 <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/"><img alt="知识共享许可协议" style="border-width:0" src="https://i.creativecommons.org/l/by-nc-sa/4.0/88x31.png" /></a><br /><span xmlns:dct="http://purl.org/dc/terms/" href="http://purl.org/dc/dcmitype/Text" property="dct:title" rel="dct:type">本教程</span> 由 <a xmlns:cc="http://creativecommons.org/ns#" href="http://book.paddlepaddle.org" property="cc:attributionName" rel="cc:attributionURL">PaddlePaddle</a> 创作，采用 <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/">知识共享 署名-非商业性使用-相同方式共享 4.0 国际 许可协议</a>进行许可。
</div> </div>
<!-- You can change the lines below now. --> <!-- You can change the lines below now. -->
......
machine_translation/index.html
浏览文件 @ 3a5ab5d9
此差异已折叠。
.tmpl/convert-markdown-into-html.sh → pre-commit-hooks/convert_markdown_into_html.py 100755 → 100644
浏览文件 @ 3a5ab5d9
markdown_file=$1 import argparse
import re
import sys
# Notice: the single-quotes around EOF below make outputs HEAD = """
# verbatium. c.f. http://stackoverflow.com/a/9870274/724872
cat <<'EOF'
<html> <html>
<head> <head>
<script type="text/x-mathjax-config"> <script type="text/x-mathjax-config">
...@@ -10,8 +10,8 @@ cat <<'EOF' ...@@ -10,8 +10,8 @@ cat <<'EOF'
extensions: ["tex2jax.js", "TeX/AMSsymbols.js", "TeX/AMSmath.js"], extensions: ["tex2jax.js", "TeX/AMSsymbols.js", "TeX/AMSmath.js"],
jax: ["input/TeX", "output/HTML-CSS"], jax: ["input/TeX", "output/HTML-CSS"],
tex2jax: { tex2jax: {
inlineMath: [ ['$','$'], ["\\(","\\)"] ], inlineMath: [ ['$','$'] ],
displayMath: [ ['$$','$$'], ["\\[","\\]"] ], displayMath: [ ['$$','$$'] ],
processEscapes: true processEscapes: true
}, },
"HTML-CSS": { availableFonts: ["TeX"] } "HTML-CSS": { availableFonts: ["TeX"] }
...@@ -44,11 +44,9 @@ cat <<'EOF' ...@@ -44,11 +44,9 @@ cat <<'EOF'
<!-- This block will be replaced by each markdown file content. Please do not change lines below.--> <!-- This block will be replaced by each markdown file content. Please do not change lines below.-->
<div id="markdown" style='display:none'> <div id="markdown" style='display:none'>
EOF """
cat $markdown_file TAIL = """
cat <<'EOF'
</div> </div>
<!-- You can change the lines below now. --> <!-- You can change the lines below now. -->
...@@ -70,4 +68,28 @@ document.getElementById("context").innerHTML = marked( ...@@ -70,4 +68,28 @@ document.getElementById("context").innerHTML = marked(
document.getElementById("markdown").innerHTML) document.getElementById("markdown").innerHTML)
</script> </script>
</body> </body>
EOF """
def convert_markdown_into_html(argv=None):
parser = argparse.ArgumentParser()
parser.add_argument('filenames', nargs='*', help='Filenames to fix')
args = parser.parse_args(argv)
retv = 0
for filename in args.filenames:
with open(
re.sub(r"README", "index", re.sub(r"\.md$", ".html", filename)),
"w") as output:
output.write(HEAD)
with open(filename) as input:
for line in input:
output.write(line)
output.write(TAIL)
return retv
if __name__ == '__main__':
sys.exit(convert_markdown_into_html())
query_relationship/index.html
浏览文件 @ 3a5ab5d9
<html> <html>
<head> <head>
<script type="text/x-mathjax-config"> <script type="text/x-mathjax-config">
...@@ -5,8 +6,8 @@ ...@@ -5,8 +6,8 @@
extensions: ["tex2jax.js", "TeX/AMSsymbols.js", "TeX/AMSmath.js"], extensions: ["tex2jax.js", "TeX/AMSsymbols.js", "TeX/AMSmath.js"],
jax: ["input/TeX", "output/HTML-CSS"], jax: ["input/TeX", "output/HTML-CSS"],
tex2jax: { tex2jax: {
inlineMath: [ ['$','$'], ["\\(","\\)"] ], inlineMath: [ ['$','$'] ],
displayMath: [ ['$$','$$'], ["\\[","\\]"] ], displayMath: [ ['$$','$$'] ],
processEscapes: true processEscapes: true
}, },
"HTML-CSS": { availableFonts: ["TeX"] } "HTML-CSS": { availableFonts: ["TeX"] }
...@@ -39,6 +40,7 @@ ...@@ -39,6 +40,7 @@
<!-- This block will be replaced by each markdown file content. Please do not change lines below.--> <!-- This block will be replaced by each markdown file content. Please do not change lines below.-->
<div id="markdown" style='display:none'> <div id="markdown" style='display:none'>
</div> </div>
<!-- You can change the lines below now. --> <!-- You can change the lines below now. -->
......
recognize_digits/index.en.html
浏览文件 @ 3a5ab5d9
<html> <html>
<head> <head>
<script type="text/x-mathjax-config"> <script type="text/x-mathjax-config">
...@@ -5,8 +6,8 @@ ...@@ -5,8 +6,8 @@
extensions: ["tex2jax.js", "TeX/AMSsymbols.js", "TeX/AMSmath.js"], extensions: ["tex2jax.js", "TeX/AMSsymbols.js", "TeX/AMSmath.js"],
jax: ["input/TeX", "output/HTML-CSS"], jax: ["input/TeX", "output/HTML-CSS"],
tex2jax: { tex2jax: {
inlineMath: [ ['$','$'], ["\\(","\\)"] ], inlineMath: [ ['$','$'] ],
displayMath: [ ['$$','$$'], ["\\[","\\]"] ], displayMath: [ ['$$','$$'] ],
processEscapes: true processEscapes: true
}, },
"HTML-CSS": { availableFonts: ["TeX"] } "HTML-CSS": { availableFonts: ["TeX"] }
...@@ -338,6 +339,7 @@ This tutorial describes a few basic Deep Learning models viz. Softmax regression ...@@ -338,6 +339,7 @@ This tutorial describes a few basic Deep Learning models viz. Softmax regression
<br/> <br/>
<a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/"><img alt="知识共享许可协议" style="border-width:0" src="https://i.creativecommons.org/l/by-nc-sa/4.0/88x31.png" /></a><br /><span xmlns:dct="http://purl.org/dc/terms/" href="http://purl.org/dc/dcmitype/Text" property="dct:title" rel="dct:type">This book</span> is created by <a xmlns:cc="http://creativecommons.org/ns#" href="http://book.paddlepaddle.org" property="cc:attributionName" rel="cc:attributionURL">PaddlePaddle</a>, and uses <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/">Shared knowledge signature - non commercial use-Sharing 4.0 International Licensing Protocal</a>. <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/"><img alt="知识共享许可协议" style="border-width:0" src="https://i.creativecommons.org/l/by-nc-sa/4.0/88x31.png" /></a><br /><span xmlns:dct="http://purl.org/dc/terms/" href="http://purl.org/dc/dcmitype/Text" property="dct:title" rel="dct:type">This book</span> is created by <a xmlns:cc="http://creativecommons.org/ns#" href="http://book.paddlepaddle.org" property="cc:attributionName" rel="cc:attributionURL">PaddlePaddle</a>, and uses <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/">Shared knowledge signature - non commercial use-Sharing 4.0 International Licensing Protocal</a>.
</div> </div>
<!-- You can change the lines below now. --> <!-- You can change the lines below now. -->
......
recognize_digits/index.html
浏览文件 @ 3a5ab5d9
<html> <html>
<head> <head>
<script type="text/x-mathjax-config"> <script type="text/x-mathjax-config">
...@@ -5,8 +6,8 @@ ...@@ -5,8 +6,8 @@
extensions: ["tex2jax.js", "TeX/AMSsymbols.js", "TeX/AMSmath.js"], extensions: ["tex2jax.js", "TeX/AMSsymbols.js", "TeX/AMSmath.js"],
jax: ["input/TeX", "output/HTML-CSS"], jax: ["input/TeX", "output/HTML-CSS"],
tex2jax: { tex2jax: {
inlineMath: [ ['$','$'], ["\\(","\\)"] ], inlineMath: [ ['$','$'] ],
displayMath: [ ['$$','$$'], ["\\[","\\]"] ], displayMath: [ ['$$','$$'] ],
processEscapes: true processEscapes: true
}, },
"HTML-CSS": { availableFonts: ["TeX"] } "HTML-CSS": { availableFonts: ["TeX"] }
...@@ -340,6 +341,7 @@ trainer.train( ...@@ -340,6 +341,7 @@ trainer.train(
<br/> <br/>
<a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/"><img alt="知识共享许可协议" style="border-width:0" src="https://i.creativecommons.org/l/by-nc-sa/4.0/88x31.png" /></a><br /><span xmlns:dct="http://purl.org/dc/terms/" href="http://purl.org/dc/dcmitype/Text" property="dct:title" rel="dct:type">本教程</span> 由 <a xmlns:cc="http://creativecommons.org/ns#" href="http://book.paddlepaddle.org" property="cc:attributionName" rel="cc:attributionURL">PaddlePaddle</a> 创作，采用 <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/">知识共享 署名-非商业性使用-相同方式共享 4.0 国际 许可协议</a>进行许可。 <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/"><img alt="知识共享许可协议" style="border-width:0" src="https://i.creativecommons.org/l/by-nc-sa/4.0/88x31.png" /></a><br /><span xmlns:dct="http://purl.org/dc/terms/" href="http://purl.org/dc/dcmitype/Text" property="dct:title" rel="dct:type">本教程</span> 由 <a xmlns:cc="http://creativecommons.org/ns#" href="http://book.paddlepaddle.org" property="cc:attributionName" rel="cc:attributionURL">PaddlePaddle</a> 创作，采用 <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/">知识共享 署名-非商业性使用-相同方式共享 4.0 国际 许可协议</a>进行许可。
</div> </div>
<!-- You can change the lines below now. --> <!-- You can change the lines below now. -->
......
recommender_system/index.en.html
浏览文件 @ 3a5ab5d9
<html> <html>
<head> <head>
<script type="text/x-mathjax-config"> <script type="text/x-mathjax-config">
...@@ -5,8 +6,8 @@ ...@@ -5,8 +6,8 @@
extensions: ["tex2jax.js", "TeX/AMSsymbols.js", "TeX/AMSmath.js"], extensions: ["tex2jax.js", "TeX/AMSsymbols.js", "TeX/AMSmath.js"],
jax: ["input/TeX", "output/HTML-CSS"], jax: ["input/TeX", "output/HTML-CSS"],
tex2jax: { tex2jax: {
inlineMath: [ ['$','$'], ["\\(","\\)"] ], inlineMath: [ ['$','$'] ],
displayMath: [ ['$$','$$'], ["\\[","\\]"] ], displayMath: [ ['$$','$$'] ],
processEscapes: true processEscapes: true
}, },
"HTML-CSS": { availableFonts: ["TeX"] } "HTML-CSS": { availableFonts: ["TeX"] }
...@@ -150,6 +151,7 @@ This tutorial goes over traditional approaches in recommender system and a deep ...@@ -150,6 +151,7 @@ This tutorial goes over traditional approaches in recommender system and a deep
<br/> <br/>
<a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/"><img alt="Creative Commons" style="border-width:0" src="https://i.creativecommons.org/l/by-nc-sa/4.0/88x31.png" /></a><br /><span xmlns:dct="http://purl.org/dc/terms/" href="http://purl.org/dc/dcmitype/Text" property="dct:title" rel="dct:type">This tutorial</span> was created by <a xmlns:cc="http://creativecommons.org/ns#" href="http://book.paddlepaddle.org" property="cc:attributionName" rel="cc:attributionURL">the PaddlePaddle community</a> and published under <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/">Common Creative 4.0 License</a>。 <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/"><img alt="Creative Commons" style="border-width:0" src="https://i.creativecommons.org/l/by-nc-sa/4.0/88x31.png" /></a><br /><span xmlns:dct="http://purl.org/dc/terms/" href="http://purl.org/dc/dcmitype/Text" property="dct:title" rel="dct:type">This tutorial</span> was created by <a xmlns:cc="http://creativecommons.org/ns#" href="http://book.paddlepaddle.org" property="cc:attributionName" rel="cc:attributionURL">the PaddlePaddle community</a> and published under <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/">Common Creative 4.0 License</a>。
</div> </div>
<!-- You can change the lines below now. --> <!-- You can change the lines below now. -->
......
recommender_system/index.html
浏览文件 @ 3a5ab5d9
此差异已折叠。
skip_thought/index.html
浏览文件 @ 3a5ab5d9
<html> <html>
<head> <head>
<script type="text/x-mathjax-config"> <script type="text/x-mathjax-config">
...@@ -5,8 +6,8 @@ ...@@ -5,8 +6,8 @@
extensions: ["tex2jax.js", "TeX/AMSsymbols.js", "TeX/AMSmath.js"], extensions: ["tex2jax.js", "TeX/AMSsymbols.js", "TeX/AMSmath.js"],
jax: ["input/TeX", "output/HTML-CSS"], jax: ["input/TeX", "output/HTML-CSS"],
tex2jax: { tex2jax: {
inlineMath: [ ['$','$'], ["\\(","\\)"] ], inlineMath: [ ['$','$'] ],
displayMath: [ ['$$','$$'], ["\\[","\\]"] ], displayMath: [ ['$$','$$'] ],
processEscapes: true processEscapes: true
}, },
"HTML-CSS": { availableFonts: ["TeX"] } "HTML-CSS": { availableFonts: ["TeX"] }
...@@ -39,6 +40,7 @@ ...@@ -39,6 +40,7 @@
<!-- This block will be replaced by each markdown file content. Please do not change lines below.--> <!-- This block will be replaced by each markdown file content. Please do not change lines below.-->
<div id="markdown" style='display:none'> <div id="markdown" style='display:none'>
</div> </div>
<!-- You can change the lines below now. --> <!-- You can change the lines below now. -->
......
speech_recognition/index.html
浏览文件 @ 3a5ab5d9
<html> <html>
<head> <head>
<script type="text/x-mathjax-config"> <script type="text/x-mathjax-config">
...@@ -5,8 +6,8 @@ ...@@ -5,8 +6,8 @@
extensions: ["tex2jax.js", "TeX/AMSsymbols.js", "TeX/AMSmath.js"], extensions: ["tex2jax.js", "TeX/AMSsymbols.js", "TeX/AMSmath.js"],
jax: ["input/TeX", "output/HTML-CSS"], jax: ["input/TeX", "output/HTML-CSS"],
tex2jax: { tex2jax: {
inlineMath: [ ['$','$'], ["\\(","\\)"] ], inlineMath: [ ['$','$'] ],
displayMath: [ ['$$','$$'], ["\\[","\\]"] ], displayMath: [ ['$$','$$'] ],
processEscapes: true processEscapes: true
}, },
"HTML-CSS": { availableFonts: ["TeX"] } "HTML-CSS": { availableFonts: ["TeX"] }
...@@ -39,6 +40,7 @@ ...@@ -39,6 +40,7 @@
<!-- This block will be replaced by each markdown file content. Please do not change lines below.--> <!-- This block will be replaced by each markdown file content. Please do not change lines below.-->
<div id="markdown" style='display:none'> <div id="markdown" style='display:none'>
</div> </div>
<!-- You can change the lines below now. --> <!-- You can change the lines below now. -->
......
understand_sentiment/index.en.html
浏览文件 @ 3a5ab5d9
<html> <html>
<head> <head>
<script type="text/x-mathjax-config"> <script type="text/x-mathjax-config">
...@@ -5,8 +6,8 @@ ...@@ -5,8 +6,8 @@
extensions: ["tex2jax.js", "TeX/AMSsymbols.js", "TeX/AMSmath.js"], extensions: ["tex2jax.js", "TeX/AMSsymbols.js", "TeX/AMSmath.js"],
jax: ["input/TeX", "output/HTML-CSS"], jax: ["input/TeX", "output/HTML-CSS"],
tex2jax: { tex2jax: {
inlineMath: [ ['$','$'], ["\\(","\\)"] ], inlineMath: [ ['$','$'] ],
displayMath: [ ['$$','$$'], ["\\[","\\]"] ], displayMath: [ ['$$','$$'] ],
processEscapes: true processEscapes: true
}, },
"HTML-CSS": { availableFonts: ["TeX"] } "HTML-CSS": { availableFonts: ["TeX"] }
...@@ -532,6 +533,7 @@ In this chapter, we use sentiment analysis as an example to introduce applying d ...@@ -532,6 +533,7 @@ In this chapter, we use sentiment analysis as an example to introduce applying d
<br/> <br/>
<a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/"><img alt="知识共享许可协议" style="border-width:0" src="https://i.creativecommons.org/l/by-nc-sa/4.0/88x31.png" /></a><br /><span xmlns:dct="http://purl.org/dc/terms/" href="http://purl.org/dc/dcmitype/Text" property="dct:title" rel="dct:type">本教程</span> 由 <a xmlns:cc="http://creativecommons.org/ns#" href="http://book.paddlepaddle.org" property="cc:attributionName" rel="cc:attributionURL">PaddlePaddle</a> 创作，采用 <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/">知识共享 署名-非商业性使用-相同方式共享 4.0 国际 许可协议</a>进行许可。 <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/"><img alt="知识共享许可协议" style="border-width:0" src="https://i.creativecommons.org/l/by-nc-sa/4.0/88x31.png" /></a><br /><span xmlns:dct="http://purl.org/dc/terms/" href="http://purl.org/dc/dcmitype/Text" property="dct:title" rel="dct:type">本教程</span> 由 <a xmlns:cc="http://creativecommons.org/ns#" href="http://book.paddlepaddle.org" property="cc:attributionName" rel="cc:attributionURL">PaddlePaddle</a> 创作，采用 <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/">知识共享 署名-非商业性使用-相同方式共享 4.0 国际 许可协议</a>进行许可。
</div> </div>
<!-- You can change the lines below now. --> <!-- You can change the lines below now. -->
......
understand_sentiment/index.html
浏览文件 @ 3a5ab5d9
<html> <html>
<head> <head>
<script type="text/x-mathjax-config"> <script type="text/x-mathjax-config">
...@@ -5,8 +6,8 @@ ...@@ -5,8 +6,8 @@
extensions: ["tex2jax.js", "TeX/AMSsymbols.js", "TeX/AMSmath.js"], extensions: ["tex2jax.js", "TeX/AMSsymbols.js", "TeX/AMSmath.js"],
jax: ["input/TeX", "output/HTML-CSS"], jax: ["input/TeX", "output/HTML-CSS"],
tex2jax: { tex2jax: {
inlineMath: [ ['$','$'], ["\\(","\\)"] ], inlineMath: [ ['$','$'] ],
displayMath: [ ['$$','$$'], ["\\[","\\]"] ], displayMath: [ ['$$','$$'] ],
processEscapes: true processEscapes: true
}, },
"HTML-CSS": { availableFonts: ["TeX"] } "HTML-CSS": { availableFonts: ["TeX"] }
...@@ -353,6 +354,7 @@ Test with Pass 0, {'classification_error_evaluator': 0.11432000249624252} ...@@ -353,6 +354,7 @@ Test with Pass 0, {'classification_error_evaluator': 0.11432000249624252}
<br/> <br/>
<a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/"><img alt="知识共享许可协议" style="border-width:0" src="https://i.creativecommons.org/l/by-nc-sa/4.0/88x31.png" /></a><br /><span xmlns:dct="http://purl.org/dc/terms/" href="http://purl.org/dc/dcmitype/Text" property="dct:title" rel="dct:type">本教程</span> 由 <a xmlns:cc="http://creativecommons.org/ns#" href="http://book.paddlepaddle.org" property="cc:attributionName" rel="cc:attributionURL">PaddlePaddle</a> 创作，采用 <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/">知识共享 署名-非商业性使用-相同方式共享 4.0 国际 许可协议</a>进行许可。 <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/"><img alt="知识共享许可协议" style="border-width:0" src="https://i.creativecommons.org/l/by-nc-sa/4.0/88x31.png" /></a><br /><span xmlns:dct="http://purl.org/dc/terms/" href="http://purl.org/dc/dcmitype/Text" property="dct:title" rel="dct:type">本教程</span> 由 <a xmlns:cc="http://creativecommons.org/ns#" href="http://book.paddlepaddle.org" property="cc:attributionName" rel="cc:attributionURL">PaddlePaddle</a> 创作，采用 <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/">知识共享 署名-非商业性使用-相同方式共享 4.0 国际 许可协议</a>进行许可。
</div> </div>
<!-- You can change the lines below now. --> <!-- You can change the lines below now. -->
......
word2vec/index.en.html
浏览文件 @ 3a5ab5d9
<html> <html>
<head> <head>
<script type="text/x-mathjax-config"> <script type="text/x-mathjax-config">
...@@ -5,8 +6,8 @@ ...@@ -5,8 +6,8 @@
extensions: ["tex2jax.js", "TeX/AMSsymbols.js", "TeX/AMSmath.js"], extensions: ["tex2jax.js", "TeX/AMSsymbols.js", "TeX/AMSmath.js"],
jax: ["input/TeX", "output/HTML-CSS"], jax: ["input/TeX", "output/HTML-CSS"],
tex2jax: { tex2jax: {
inlineMath: [ ['$','$'], ["\\(","\\)"] ], inlineMath: [ ['$','$'] ],
displayMath: [ ['$$','$$'], ["\\[","\\]"] ], displayMath: [ ['$$','$$'] ],
processEscapes: true processEscapes: true
}, },
"HTML-CSS": { availableFonts: ["TeX"] } "HTML-CSS": { availableFonts: ["TeX"] }
...@@ -219,6 +220,7 @@ In information retrieval, the relevance between the query and document keyword c ...@@ -219,6 +220,7 @@ In information retrieval, the relevance between the query and document keyword c
<br/> <br/>
<a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/"><img alt="知识共享许可协议" style="border-width:0" src="https://i.creativecommons.org/l/by-nc-sa/4.0/88x31.png" /></a><br /><span xmlns:dct="http://purl.org/dc/terms/" href="http://purl.org/dc/dcmitype/Text" property="dct:title" rel="dct:type">本教程</span> 由 <a xmlns:cc="http://creativecommons.org/ns#" href="http://book.paddlepaddle.org" property="cc:attributionName" rel="cc:attributionURL">PaddlePaddle</a> 创作，采用 <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/">知识共享 署名-非商业性使用-相同方式共享 4.0 国际 许可协议</a>进行许可。 <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/"><img alt="知识共享许可协议" style="border-width:0" src="https://i.creativecommons.org/l/by-nc-sa/4.0/88x31.png" /></a><br /><span xmlns:dct="http://purl.org/dc/terms/" href="http://purl.org/dc/dcmitype/Text" property="dct:title" rel="dct:type">本教程</span> 由 <a xmlns:cc="http://creativecommons.org/ns#" href="http://book.paddlepaddle.org" property="cc:attributionName" rel="cc:attributionURL">PaddlePaddle</a> 创作，采用 <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/">知识共享 署名-非商业性使用-相同方式共享 4.0 国际 许可协议</a>进行许可。
</div> </div>
<!-- You can change the lines below now. --> <!-- You can change the lines below now. -->
......
word2vec/index.html
浏览文件 @ 3a5ab5d9
<html> <html>
<head> <head>
<script type="text/x-mathjax-config"> <script type="text/x-mathjax-config">
...@@ -5,8 +6,8 @@ ...@@ -5,8 +6,8 @@
extensions: ["tex2jax.js", "TeX/AMSsymbols.js", "TeX/AMSmath.js"], extensions: ["tex2jax.js", "TeX/AMSsymbols.js", "TeX/AMSmath.js"],
jax: ["input/TeX", "output/HTML-CSS"], jax: ["input/TeX", "output/HTML-CSS"],
tex2jax: { tex2jax: {
inlineMath: [ ['$','$'], ["\\(","\\)"] ], inlineMath: [ ['$','$'] ],
displayMath: [ ['$$','$$'], ["\\[","\\]"] ], displayMath: [ ['$$','$$'] ],
processEscapes: true processEscapes: true
}, },
"HTML-CSS": { availableFonts: ["TeX"] } "HTML-CSS": { availableFonts: ["TeX"] }
...@@ -39,6 +40,7 @@ ...@@ -39,6 +40,7 @@
<!-- This block will be replaced by each markdown file content. Please do not change lines below.--> <!-- This block will be replaced by each markdown file content. Please do not change lines below.-->
<div id="markdown" style='display:none'> <div id="markdown" style='display:none'>
# 词向量 # 词向量
本教程源代码目录在[book/word2vec](https://github.com/PaddlePaddle/book/tree/develop/word2vec)， 初次使用请参考PaddlePaddle[安装教程](http://www.paddlepaddle.org/doc_cn/build_and_install/index.html)。 本教程源代码目录在[book/word2vec](https://github.com/PaddlePaddle/book/tree/develop/word2vec)， 初次使用请参考PaddlePaddle[安装教程](http://www.paddlepaddle.org/doc_cn/build_and_install/index.html)。
...@@ -266,38 +268,38 @@ def wordemb(inlayer): ...@@ -266,38 +268,38 @@ def wordemb(inlayer):
- 定义输入层接受的数据类型以及名字。 - 定义输入层接受的数据类型以及名字。
```python ```python
def main(): paddle.init(use_gpu=False, trainer_count=3) # 初始化PaddlePaddle
paddle.init(use_gpu=False, trainer_count=1) # 初始化PaddlePaddle word_dict = paddle.dataset.imikolov.build_dict()
word_dict = paddle.dataset.imikolov.build_dict() dict_size = len(word_dict)
dict_size = len(word_dict) # 每个输入层都接受整形数据，这些数据的范围是[0, dict_size)
# 每个输入层都接受整形数据，这些数据的范围是[0, dict_size) firstword = paddle.layer.data(
firstword = paddle.layer.data(
name="firstw", type=paddle.data_type.integer_value(dict_size)) name="firstw", type=paddle.data_type.integer_value(dict_size))
secondword = paddle.layer.data( secondword = paddle.layer.data(
name="secondw", type=paddle.data_type.integer_value(dict_size)) name="secondw", type=paddle.data_type.integer_value(dict_size))
thirdword = paddle.layer.data( thirdword = paddle.layer.data(
name="thirdw", type=paddle.data_type.integer_value(dict_size)) name="thirdw", type=paddle.data_type.integer_value(dict_size))
fourthword = paddle.layer.data( fourthword = paddle.layer.data(
name="fourthw", type=paddle.data_type.integer_value(dict_size)) name="fourthw", type=paddle.data_type.integer_value(dict_size))
nextword = paddle.layer.data( nextword = paddle.layer.data(
name="fifthw", type=paddle.data_type.integer_value(dict_size)) name="fifthw", type=paddle.data_type.integer_value(dict_size))
Efirst = wordemb(firstword) Efirst = wordemb(firstword)
Esecond = wordemb(secondword) Esecond = wordemb(secondword)
Ethird = wordemb(thirdword) Ethird = wordemb(thirdword)
Efourth = wordemb(fourthword) Efourth = wordemb(fourthword)
``` ```
- 将这n-1个词向量经过concat_layer连接成一个大向量作为历史文本特征。 - 将这n-1个词向量经过concat_layer连接成一个大向量作为历史文本特征。
```python ```python
contextemb = paddle.layer.concat(input=[Efirst, Esecond, Ethird, Efourth]) contextemb = paddle.layer.concat(input=[Efirst, Esecond, Ethird, Efourth])
``` ```
- 将历史文本特征经过一个全连接得到文本隐层特征。 - 将历史文本特征经过一个全连接得到文本隐层特征。
```python ```python
hidden1 = paddle.layer.fc(input=contextemb, hidden1 = paddle.layer.fc(input=contextemb,
size=hiddensize, size=hiddensize,
act=paddle.activation.Sigmoid(), act=paddle.activation.Sigmoid(),
layer_attr=paddle.attr.Extra(drop_rate=0.5), layer_attr=paddle.attr.Extra(drop_rate=0.5),
...@@ -310,7 +312,7 @@ def main(): ...@@ -310,7 +312,7 @@ def main():
- 将文本隐层特征，再经过一个全连接，映射成一个$|V|$维向量，同时通过softmax归一化得到这`|V|`个词的生成概率。 - 将文本隐层特征，再经过一个全连接，映射成一个$|V|$维向量，同时通过softmax归一化得到这`|V|`个词的生成概率。
```python ```python
predictword = paddle.layer.fc(input=hidden1, predictword = paddle.layer.fc(input=hidden1,
size=dict_size, size=dict_size,
bias_attr=paddle.attr.Param(learning_rate=2), bias_attr=paddle.attr.Param(learning_rate=2),
act=paddle.activation.Softmax()) act=paddle.activation.Softmax())
...@@ -329,11 +331,11 @@ cost = paddle.layer.classification_cost(input=predictword, label=nextword) ...@@ -329,11 +331,11 @@ cost = paddle.layer.classification_cost(input=predictword, label=nextword)
- 正则化（regularization）： 是防止网络过拟合的一种手段，此处采用L2正则化。 - 正则化（regularization）： 是防止网络过拟合的一种手段，此处采用L2正则化。
```python ```python
parameters = paddle.parameters.create(cost) parameters = paddle.parameters.create(cost)
adam_optimizer = paddle.optimizer.Adam( adam_optimizer = paddle.optimizer.Adam(
learning_rate=3e-3, learning_rate=3e-3,
regularization=paddle.optimizer.L2Regularization(8e-4)) regularization=paddle.optimizer.L2Regularization(8e-4))
trainer = paddle.trainer.SGD(cost, parameters, adam_optimizer) trainer = paddle.trainer.SGD(cost, parameters, adam_optimizer)
``` ```
下一步，我们开始训练过程。`paddle.dataset.imikolov.train()`和`paddle.dataset.imikolov.test()`分别做训练和测试数据集。这两个函数各自返回一个reader——PaddlePaddle中的reader是一个Python函数，每次调用的时候返回一个Python generator。 下一步，我们开始训练过程。`paddle.dataset.imikolov.train()`和`paddle.dataset.imikolov.test()`分别做训练和测试数据集。这两个函数各自返回一个reader——PaddlePaddle中的reader是一个Python函数，每次调用的时候返回一个Python generator。
...@@ -341,112 +343,94 @@ cost = paddle.layer.classification_cost(input=predictword, label=nextword) ...@@ -341,112 +343,94 @@ cost = paddle.layer.classification_cost(input=predictword, label=nextword)
`paddle.batch`的输入是一个reader，输出是一个batched reader —— 在PaddlePaddle里，一个reader每次yield一条训练数据，而一个batched reader每次yield一个minbatch。 `paddle.batch`的输入是一个reader，输出是一个batched reader —— 在PaddlePaddle里，一个reader每次yield一条训练数据，而一个batched reader每次yield一个minbatch。
```python ```python
def event_handler(event): import gzip
def event_handler(event):
if isinstance(event, paddle.event.EndIteration): if isinstance(event, paddle.event.EndIteration):
if event.batch_id % 100 == 0: if event.batch_id % 100 == 0:
print "Pass %d, Batch %d, Cost %f, %s" % (
event.pass_id, event.batch_id, event.cost, event.metrics)
if isinstance(event, paddle.event.EndPass):
result = trainer.test( result = trainer.test(
paddle.batch( paddle.batch(
paddle.dataset.imikolov.test(word_dict, N), 32)) paddle.dataset.imikolov.test(word_dict, N), 32))
print "Pass %d, Batch %d, Cost %f, %s, Testing metrics %s" % ( print "Pass %d, Testing metrics %s" % (event.pass_id, result.metrics)
event.pass_id, event.batch_id, event.cost, event.metrics, with gzip.open("model_%d.tar.gz"%event.pass_id, 'w') as f:
result.metrics) parameters.to_tar(f)
trainer.train( trainer.train(
paddle.batch(paddle.dataset.imikolov.train(word_dict, N), 32), paddle.batch(paddle.dataset.imikolov.train(word_dict, N), 32),
num_passes=30, num_passes=100,
event_handler=event_handler) event_handler=event_handler)
``` ```
训练过程是完全自动的，event_handler里打印的日志类似如下所示： ...
Pass 0, Batch 25000, Cost 4.251861, {'classification_error_evaluator': 0.84375}
Pass 0, Batch 25100, Cost 4.847692, {'classification_error_evaluator': 0.8125}
Pass 0, Testing metrics {'classification_error_evaluator': 0.7417652606964111}
训练过程是完全自动的，event_handler里打印的日志类似如上所示：
```text
.............................
I1222 09:27:16.477841 12590 TrainerInternal.cpp:162] Batch=3000 samples=300000 AvgCost=5.36135 CurrentCost=5.36135 Eval: classification_error_evaluator=0.818653 CurrentEval: class
ification_error_evaluator=0.818653
.............................
I1222 09:27:22.416700 12590 TrainerInternal.cpp:162] Batch=6000 samples=600000 AvgCost=5.29301 CurrentCost=5.22467 Eval: classification_error_evaluator=0.814542 CurrentEval: class
ification_error_evaluator=0.81043
.............................
I1222 09:27:28.343756 12590 TrainerInternal.cpp:162] Batch=9000 samples=900000 AvgCost=5.22494 CurrentCost=5.08876 Eval: classification_error_evaluator=0.810088 CurrentEval: class
ification_error_evaluator=0.80118
..I1222 09:27:29.128582 12590 TrainerInternal.cpp:179] Pass=0 Batch=9296 samples=929600 AvgCost=5.21786 Eval: classification_error_evaluator=0.809647
I1222 09:27:29.627616 12590 Tester.cpp:111] Test samples=73760 cost=4.9594 Eval: classification_error_evaluator=0.79676
I1222 09:27:29.627713 12590 GradientMachine.cpp:112] Saving parameters to model/pass-00000
```
经过30个pass，我们将得到平均错误率为classification_error_evaluator=0.735611。 经过30个pass，我们将得到平均错误率为classification_error_evaluator=0.735611。
## 应用模型 ## 应用模型
训练模型后，我们可以加载模型参数，用训练出来的词向量初始化其他模型，也可以将模型参数从二进制格式转换成文本格式进行后续应用。 训练模型后，我们可以加载模型参数，用训练出来的词向量初始化其他模型，也可以将模型查看参数用来做后续应用。
### 初始化其他模型
训练好的模型参数可以用来初始化其他模型。具体方法如下：
在PaddlePaddle 训练命令行中，用`--init_model_path` 来定义初始化模型的位置，用`--load_missing_parameter_strategy`指定除了词向量以外的新模型其他参数的初始化策略。注意，新模型需要和原模型共享被初始化参数的参数名。
### 查看词向量 ### 查看词向量
PaddlePaddle训练出来的参数为二进制格式，存储在对应训练pass的文件夹下。这里我们提供了文件`format_convert.py`用来互转PaddlePaddle训练结果的二进制文件和文本格式特征文件。
```bash PaddlePaddle训练出来的参数可以直接使用`parameters.get()`获取出来。例如查看单词的word的词向量，即为
python format_convert.py --b2t -i INPUT -o OUTPUT -d DIM
```
其中，INPUT是输入的（二进制）词向量模型名称，OUTPUT是输出的文本模型名称，DIM是词向量参数维度。
用法如：
```bash ```python
python format_convert.py --b2t -i model/pass-00029/_proj -o model/pass-00029/_proj.txt -d 32 embeddings = parameters.get("_proj").reshape(len(word_dict), embsize)
```
转换后得到的文本文件如下：
```text print embeddings[word_dict['word']]
0,4,62496
-0.7444070,-0.1846171,-1.5771370,0.7070392,2.1963732,-0.0091410, ......
-0.0721337,-0.2429973,-0.0606297,0.1882059,-0.2072131,-0.7661019, ......
......
``` ```
其中，第一行是PaddlePaddle 输出文件的格式说明，包含3个属性：<br/> [-0.38961065 -0.02392169 -0.00093231 0.36301503 0.13538605 0.16076435
1) PaddlePaddle的版本号，本例中为0;<br/> -0.0678709 0.1090285 0.42014077 -0.24119169 -0.31847557 0.20410083
2) 浮点数占用的字节数，本例中为4;<br/> 0.04910378 0.19021918 -0.0122014 -0.04099389 -0.16924137 0.1911236
3) 总计的参数个数, 本例中为62496（即1953*32）;<br/> -0.10917275 0.13068172 -0.23079982 0.42699069 -0.27679482 -0.01472992
第二行及之后的每一行都按顺序表示字典里一个词的特征，用逗号分隔。 0.2069038 0.09005053 -0.3282454 0.12717034 -0.24218646 0.25304323
0.19072419 -0.24286366]
### 修改词向量
我们可以对词向量进行修改，并转换成PaddlePaddle参数二进制格式，方法：
```bash ### 修改词向量
python format_convert.py --t2b -i INPUT -o OUTPUT
```
其中，INPUT是输入的输入的文本词向量模型名称，OUTPUT是输出的二进制词向量模型名称 获得到的embedding为一个标准的numpy矩阵。我们可以对这个numpy矩阵进行修改，然后赋值回去。
输入的文本格式如下（注意，不包含上面二进制转文本后第一行的格式说明）：
```text
-0.7444070,-0.1846171,-1.5771370,0.7070392,2.1963732,-0.0091410, ......
-0.0721337,-0.2429973,-0.0606297,0.1882059,-0.2072131,-0.7661019, ......
......
```
```python
def modify_embedding(emb):
# Add your modification here.
pass
modify_embedding(embeddings)
parameters.set("_proj", embeddings)
```
### 计算词语之间的余弦距离 ### 计算词语之间的余弦距离
两个向量之间的距离可以用余弦值来表示，余弦值在$[-1,1]$的区间内，向量间余弦值越大，其距离越近。这里我们在`calculate_dis.py`中实现不同词语的距离度量。 两个向量之间的距离可以用余弦值来表示，余弦值在$[-1,1]$的区间内，向量间余弦值越大，其距离越近。这里我们在`calculate_dis.py`中实现不同词语的距离度量。
用法如下： 用法如下：
```bash
python calculate_dis.py VOCABULARY EMBEDDINGLAYER`
```
其中，`VOCABULARY`是字典，`EMBEDDINGLAYER`是词向量模型，示例如下： ```python
from scipy import spatial
emb_1 = embeddings[word_dict['world']]
emb_2 = embeddings[word_dict['would']]
```bash print spatial.distance.cosine(emb_1, emb_2)
python calculate_dis.py data/vocabulary.txt model/pass-00029/_proj.txt
``` ```
0.99375076448
## 总结 ## 总结
本章中，我们介绍了词向量、语言模型和词向量的关系、以及如何通过训练神经网络模型获得词向量。在信息检索中，我们可以根据向量间的余弦夹角，来判断query和文档关键词这二者间的相关性。在句法分析和语义分析中，训练好的词向量可以用来初始化模型，以得到更好的效果。在文档分类中，有了词向量之后，可以用聚类的方法将文档中同义词进行分组。希望大家在本章后能够自行运用词向量进行相关领域的研究。 本章中，我们介绍了词向量、语言模型和词向量的关系、以及如何通过训练神经网络模型获得词向量。在信息检索中，我们可以根据向量间的余弦夹角，来判断query和文档关键词这二者间的相关性。在句法分析和语义分析中，训练好的词向量可以用来初始化模型，以得到更好的效果。在文档分类中，有了词向量之后，可以用聚类的方法将文档中同义词进行分组。希望大家在本章后能够自行运用词向量进行相关领域的研究。
...@@ -461,6 +445,7 @@ python calculate_dis.py data/vocabulary.txt model/pass-00029/_proj.txt ...@@ -461,6 +445,7 @@ python calculate_dis.py data/vocabulary.txt model/pass-00029/_proj.txt
<br/> <br/>
<a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/"><img alt="知识共享许可协议" style="border-width:0" src="https://i.creativecommons.org/l/by-nc-sa/4.0/88x31.png" /></a><br /><span xmlns:dct="http://purl.org/dc/terms/" href="http://purl.org/dc/dcmitype/Text" property="dct:title" rel="dct:type">本教程</span> 由 <a xmlns:cc="http://creativecommons.org/ns#" href="http://book.paddlepaddle.org" property="cc:attributionName" rel="cc:attributionURL">PaddlePaddle</a> 创作，采用 <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/">知识共享 署名-非商业性使用-相同方式共享 4.0 国际 许可协议</a>进行许可。 <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/"><img alt="知识共享许可协议" style="border-width:0" src="https://i.creativecommons.org/l/by-nc-sa/4.0/88x31.png" /></a><br /><span xmlns:dct="http://purl.org/dc/terms/" href="http://purl.org/dc/dcmitype/Text" property="dct:title" rel="dct:type">本教程</span> 由 <a xmlns:cc="http://creativecommons.org/ns#" href="http://book.paddlepaddle.org" property="cc:attributionName" rel="cc:attributionURL">PaddlePaddle</a> 创作，采用 <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/">知识共享 署名-非商业性使用-相同方式共享 4.0 国际 许可协议</a>进行许可。
</div> </div>
<!-- You can change the lines below now. --> <!-- You can change the lines below now. -->
......
Markdown is supported
0% 或 .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑！
想要评论请 注册 或