diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index d5932484f3513368ae164390c55eed85b61424a6..4060c19ca8b92e43aa66975f2375da34aaab377e 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -1,10 +1,10 @@ -- repo: https://github.com/reyoung/mirrors-yapf.git - sha: v0.13.2 +- repo: https://github.com/pre-commit/mirrors-yapf.git + sha: v0.16.0 hooks: - - id: yapf - files: (.*\.(py|bzl)|BUILD|.*\.BUILD|WORKSPACE)$ # Bazel BUILD files follow Python syntax. + - id: yapf + files: \.py$ - repo: https://github.com/pre-commit/pre-commit-hooks - sha: v0.7.1 + sha: a11d9314b22d8f8c7556443875b731ef05965464 hooks: - id: check-merge-conflict - id: check-symlinks @@ -24,16 +24,17 @@ files: \.md$ - id: remove-tabs files: \.md$ -- repo: git://github.com/reyoung/pre-commit-hooks-jinja-compile.git - sha: 85ad800cbc9c60a64230d60971aa9576fd57e508 - hooks: - - id: convert-jinja2-into-html -- repo: local - hooks: - - id: convert-markdown-into-html - name: convert-markdown-into-html - description: "Convert README.md into index.html and README.en.md into index.en.html" - entry: python pre-commit-hooks/convert_markdown_into_html.py - language: system - files: .+README(\.en)?\.md$ - +- repo: local + hooks: + - id: convert-markdown-into-html + name: convert-markdown-into-html + description: Convert README.md into index.html and README.en.md into index.en.html + entry: python pre-commit-hooks/convert_markdown_into_html.py + language: system + files: .+README(\.en)?\.md$ + - id: convert-markdown-into-ipynb + name: convert-markdown-into-ipynb + description: Convert README.md into README.ipynb and README.en.md into README.en.ipynb + entry: ./pre-commit-hooks/convert_markdown_into_ipynb.sh + language: system + files: .+README(\.en)?\.md$ diff --git a/.travis.yml b/.travis.yml index dcd438ed29dedafe334957bf0f0dd7cf4c2090cc..b4b1e0f9ae819923373736317b77c75787d0d39b 100644 --- a/.travis.yml +++ b/.travis.yml @@ -14,8 +14,10 @@ addons: - python - python-pip - python2.7-dev + - golang before_install: - pip install -U virtualenv pre-commit pip + - GOPATH=/tmp/go go get -u github.com/wangkuiyi/ipynb/markdown-to-ipynb script: - travis/precommit.sh notifications: diff --git a/README.en.md b/README.en.md new file mode 100644 index 0000000000000000000000000000000000000000..6e235bd13e8c6c772d0e166506d68572288d3772 --- /dev/null +++ b/README.en.md @@ -0,0 +1,12 @@ +# Deep Learning with PaddlePaddle + +1. [Fit a Line](http://book.paddlepaddle.org/fit_a_line/index.en.html) +1. [Recognize Digits](http://book.paddlepaddle.org/recognize_digits/index.en.html) +1. [Image Classification](http://book.paddlepaddle.org/image_classification/index.en.html) +1. [Word to Vector](http://book.paddlepaddle.org/word2vec/index.en.html) +1. [Understand Sentiment](http://book.paddlepaddle.org/understand_sentiment/index.en.html) +1. [Label Semantic Roles](http://book.paddlepaddle.org/label_semantic_roles/index.en.html) +1. [Machine Translation](http://book.paddlepaddle.org/machine_translation/index.en.html) +1. [Recommender System](http://book.paddlepaddle.org/recommender_system/index.en.html) + +This tutorial is contributed by PaddlePaddle, and licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. diff --git a/README.md b/README.md index 459bfbaefd59deb97b87e3dba80e26e0157c85ce..2aae8fa2f2a1db95b566665c99b20846cbcbba07 100644 --- a/README.md +++ b/README.md @@ -1,27 +1,13 @@ # 深度学习入门 -1. [新手入门](fit_a_line/) [[html](http://book.paddlepaddle.org/fit_a_line)] -1. [识别数字](recognize_digits/) [[html](http://book.paddlepaddle.org/recognize_digits)] -1. [图像分类](image_classification/) [[html](http://book.paddlepaddle.org/image_classification)] -1. [词向量](word2vec/) [[html](http://book.paddlepaddle.org/word2vec)] -1. [情感分析](understand_sentiment/) [[html](http://book.paddlepaddle.org/understand_sentiment)] -1. [语义角色标注](label_semantic_roles/) [[html](http://book.paddlepaddle.org/label_semantic_roles)] -1. [机器翻译](machine_translation/) [[html](http://book.paddlepaddle.org/machine_translation)] -1. [个性化推荐](recommender_system/) [[html](http://book.paddlepaddle.org/recommender_system)] +1. [新手入门](http://book.paddlepaddle.org/fit_a_line) +1. [识别数字](http://book.paddlepaddle.org/recognize_digits) +1. [图像分类](http://book.paddlepaddle.org/image_classification) +1. [词向量](http://book.paddlepaddle.org/word2vec) +1. [情感分析](http://book.paddlepaddle.org/understand_sentiment) +1. [语义角色标注](http://book.paddlepaddle.org/label_semantic_roles) +1. [机器翻译](http://book.paddlepaddle.org/machine_translation) +1. [个性化推荐](http://book.paddlepaddle.org/recommender_system) - -# Deep Learning Introduction - -1. [Fit a Line](fit_a_line/) [[html](http://book.paddlepaddle.org/fit_a_line/index.en.html)] -1. [Recognize Digits](recognize_digits/) [[html](http://book.paddlepaddle.org/recognize_digits/index.en.html)] -1. [Image Classification](image_classification/) [[html](http://book.paddlepaddle.org/image_classification/index.en.html)] -1. [Word to Vector](word2vec/) [[html](http://book.paddlepaddle.org/word2vec/index.en.html)] -1. [Understand Sentiment](understand_sentiment/) [[html](http://book.paddlepaddle.org/understand_sentiment/index.en.html)] -1. [Label Semantic Roles](label_semantic_roles/) [[html](http://book.paddlepaddle.org/label_semantic_roles/index.en.html)] -1. [Machine Translation](machine_translation/) [[html](http://book.paddlepaddle.org/machine_translation/index.en.html)] -1. [Recommender System](recommender_system/) [[html](http://book.paddlepaddle.org/recommender_system/index.en.html)] - -
知识共享许可协议
本教程 由 PaddlePaddle 创作，采用 知识共享 署名-非商业性使用-相同方式共享 4.0 国际 许可协议进行许可。 -This tutorial is contributed by PaddlePaddle, and licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. diff --git a/fit_a_line/README.en.ipynb b/fit_a_line/README.en.ipynb new file mode 100644 index 0000000000000000000000000000000000000000..110b3031026f1fd0d65cbb1b3a7d45c43c86ee7a --- /dev/null +++ b/fit_a_line/README.en.ipynb @@ -0,0 +1,412 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Linear Regression\n", + "Let us begin the tutorial with a classical problem called Linear Regression \\[[1](#References)\\]. In this chapter, we will train a model from a realistic dataset to predict home prices. Some important concepts in Machine Learning will be covered through this example.\n", + "\n", + "The source code for this tutorial lives on [book/fit_a_line](https://github.com/PaddlePaddle/book/tree/develop/fit_a_line). For instructions on getting started with PaddlePaddle, see [PaddlePaddle installation guide](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/getstarted/build_and_install/docker_install_en.rst).\n", + "\n", + "## Problem Setup\n", + "Suppose we have a dataset of $n$ real estate properties. These real estate properties will be referred to as *homes* in this chapter for clarity.\n", + "\n", + "Each home is associated with $d$ attributes. The attributes describe characteristics such the number of rooms in the home, the number of schools or hospitals in the neighborhood, and the traffic condition nearby.\n", + "\n", + "In our problem setup, the attribute $x_{i,j}$ denotes the $j$th characteristic of the $i$th home. In addition, $y_i$ denotes the price of the $i$th home. Our task is to predict $y_i$ given a set of attributes $\\{x_{i,1}, ..., x_{i,d}\\}$. We assume that the price of a home is a linear combination of all of its attributes, namely,\n", + "\n", + "$$y_i = \\omega_1x_{i,1} + \\omega_2x_{i,2} + \\ldots + \\omega_dx_{i,d} + b, i=1,\\ldots,n$$\n", + "\n", + "where $\\vec{\\omega}$ and $b$ are the model parameters we want to estimate. Once they are learned, we will be able to predict the price of a home, given the attributes associated with it. We call this model **Linear Regression**. In other words, we want to regress a value against several values linearly. In practice, a linear model is often too simplistic to capture the real relationships between the variables. Yet, because Linear Regression is easy to train and analyze, it has been applied to a large number of real problems. As a result, it is an important topic in many classic Statistical Learning and Machine Learning textbooks \\[[2,3,4](#References)\\].\n", + "\n", + "## Results Demonstration\n", + "We first show the result of our model. The dataset [UCI Housing Data Set](https://archive.ics.uci.edu/ml/datasets/Housing) is used to train a linear model to predict the home prices in Boston. The figure below shows the predictions the model makes for some home prices. The $X$-axis represents the median value of the prices of simlilar homes within a bin, while the $Y$-axis represents the home value our linear model predicts. The dotted line represents points where $X=Y$. When reading the diagram, the more precise the model predicts, the closer the point is to the dotted line.\n", + "\u003cp align=\"center\"\u003e\n", + " \u003cimg src = \"image/predictions_en.png\" width=400\u003e\u003cbr/\u003e\n", + " Figure 1. Predicted Value V.S. Actual Value\n", + "\u003c/p\u003e\n", + "\n", + "## Model Overview\n", + "\n", + "### Model Definition\n", + "\n", + "In the UCI Housing Data Set, there are 13 home attributes $\\{x_{i,j}\\}$ that are related to the median home price $y_i$, which we aim to predict. Thus, our model can be written as:\n", + "\n", + "$$\\hat{Y} = \\omega_1X_{1} + \\omega_2X_{2} + \\ldots + \\omega_{13}X_{13} + b$$\n", + "\n", + "where $\\hat{Y}$ is the predicted value used to differentiate from actual value $Y$. The model learns parameters $\\omega_1, \\ldots, \\omega_{13}, b$, where the entries of $\\vec{\\omega}$ are **weights** and $b$ is **bias**.\n", + "\n", + "Now we need an objective to optimize, so that the learned parameters can make $\\hat{Y}$ as close to $Y$ as possible. Let's refer to the concept of [Loss Function (Cost Function)](https://en.wikipedia.org/wiki/Loss_function). A loss function must output a non-negative value, given any pair of the actual value $y_i$ and the predicted value $\\hat{y_i}$. This value reflects the magnitutude of the model error.\n", + "\n", + "For Linear Regression, the most common loss function is [Mean Square Error (MSE)](https://en.wikipedia.org/wiki/Mean_squared_error) which has the following form:\n", + "\n", + "$$MSE=\\frac{1}{n}\\sum_{i=1}^{n}{(\\hat{Y_i}-Y_i)}^2$$\n", + "\n", + "That is, for a dataset of size $n$, MSE is the average value of the the prediction sqaure errors.\n", + "\n", + "### Training\n", + "\n", + "After setting up our model, there are several major steps to go through to train it:\n", + "1. Initialize the parameters including the weights $\\vec{\\omega}$ and the bias $b$. For example, we can set their mean values as $0$s, and their standard deviations as $1$s.\n", + "2. Feedforward. Evaluate the network output and compute the corresponding loss.\n", + "3. [Backpropagate](https://en.wikipedia.org/wiki/Backpropagation) the errors. The errors will be propagated from the output layer back to the input layer, during which the model parameters will be updated with the corresponding errors.\n", + "4. Repeat steps 2~3, until the loss is below a predefined threshold or the maximum number of repeats is reached.\n", + "\n", + "## Dataset\n", + "\n", + "### Python Dataset Modules\n", + "\n", + "Our program starts with importing necessary packages:\n", + "\n" + ] + }, + { + "cell_type": "code", + "metadata": { + "editable": true + }, + "source": [ + "import paddle.v2 as paddle\n", + "import paddle.v2.dataset.uci_housing as uci_housing\n" + ], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n" + ] + } + ], + "execution_count": 1 + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "We encapsulated the [UCI Housing Data Set](https://archive.ics.uci.edu/ml/datasets/Housing) in our Python module `uci_housing`. This module can\n", + "\n", + "1. download the dataset to `~/.cache/paddle/dataset/uci_housing/housing.data`, if not yet, and\n", + "2. [preprocesses](#preprocessing) the dataset.\n", + "\n", + "### An Introduction of the Dataset\n", + "\n", + "The UCI housing dataset has 506 instances. Each instance describes the attributes of a house in surburban Boston. The attributes are explained below:\n", + "\n", + "| Attribute Name | Characteristic | Data Type |\n", + "| ------| ------ | ------ |\n", + "| CRIM | per capita crime rate by town | Continuous|\n", + "| ZN | proportion of residential land zoned for lots over 25,000 sq.ft. | Continuous |\n", + "| INDUS | proportion of non-retail business acres per town | Continuous |\n", + "| CHAS | Charles River dummy variable | Discrete, 1 if tract bounds river; 0 otherwise|\n", + "| NOX | nitric oxides concentration (parts per 10 million) | Continuous |\n", + "| RM | average number of rooms per dwelling | Continuous |\n", + "| AGE | proportion of owner-occupied units built prior to 1940 | Continuous |\n", + "| DIS | weighted distances to five Boston employment centres | Continuous |\n", + "| RAD | index of accessibility to radial highways | Continuous |\n", + "| TAX | full-value property-tax rate per $10,000 | Continuous |\n", + "| PTRATIO | pupil-teacher ratio by town | Continuous |\n", + "| B | 1000(Bk - 0.63)^2 where Bk is the proportion of blacks by town | Continuous |\n", + "| LSTAT | % lower status of the population | Continuous |\n", + "| MEDV | Median value of owner-occupied homes in $1000's | Continuous |\n", + "\n", + "The last entry is the median home price.\n", + "\n", + "### Preprocessing\n", + "#### Continuous and Discrete Data\n", + "We define a feature vector of length 13 for each home, where each entry corresponds to an attribute. Our first observation is that, among the 13 dimensions, there are 12 continuous dimensions and 1 discrete dimension.\n", + "\n", + "Note that although a discrete value is also written as numeric values such as 0, 1, or 2, its meaning differs from a continuous value drastically. The linear difference between two discrete values has no meaning. For example, suppose $0$, $1$, and $2$ are used to represent colors *Red*, *Green*, and *Blue* respectively. Judging from the numeric representation of these colors, *Red* differs more from *Blue* than it does from *Green*. Yet in actuality, it is not true that extent to which the color *Blue* is different from *Red* is greater than the extent to which *Green* is different from *Red*. Therefore, when handling a discrete feature that has $d$ possible values, we usually convert it to $d$ new features where each feature takes a binary value, $0$ or $1$, indicating whether the original value is absent or present. Alternatively, the discrete features can be mapped onto a continuous multi-dimensional vector through an embedding table. For our problem here, because CHAS itself is a binary discrete value, we do not need to do any preprocessing.\n", + "\n", + "#### Feature Normalization\n", + "We also observe a huge difference among the value ranges of the 13 features (Figure 2). For instance, the values of feature *B* fall in $[0.32, 396.90]$, whereas those of feature *NOX* has a range of $[0.3850, 0.8170]$. An effective optimization would require data normalization. The goal of data normalization is to scale te values of each feature into roughly the same range, perhaps $[-0.5, 0.5]$. Here, we adopt a popular normalization technique where we substract the mean value from the feature value and divide the result by the width of the original range.\n", + "\n", + "There are at least three reasons for [Feature Normalization](https://en.wikipedia.org/wiki/Feature_scaling) (Feature Scaling):\n", + "- A value range that is too large or too small might cause floating number overflow or underflow during computation.\n", + "- Different value ranges might result in varying *importances* of different features to the model (at least in the beginning of the training process). This assumption about the data is often unreasonable, making the optimization difficult, which in turn results in increased training time.\n", + "- Many machine learning techniques or models (e.g., *L1/L2 regularization* and *Vector Space Model*) assumes that all the features have roughly zero means and their value ranges are similar.\n", + "\n", + "\u003cp align=\"center\"\u003e\n", + " \u003cimg src = \"image/ranges_en.png\" width=550\u003e\u003cbr/\u003e\n", + " Figure 2. The value ranges of the features\n", + "\u003c/p\u003e\n", + "\n", + "#### Prepare Training and Test Sets\n", + "We split the dataset in two, one for adjusting the model parameters, namely, for model training, and the other for model testing. The model error on the former is called the **training error**, and the error on the latter is called the **test error**. Our goal in training a model is to find the statistical dependency between the outputs and the inputs, so that we can predict new outputs given new inputs. As a result, the test error reflects the performance of the model better than the training error does. We consider two things when deciding the ratio of the training set to the test set: 1) More training data will decrease the variance of the parameter estimation, yielding more reliable models; 2) More test data will decrease the variance of the test error, yielding more reliable test errors. One standard split ratio is $8:2$.\n", + "\n", + "\n", + "When training complex models, we usually have one more split: the validation set. Complex models usually have [Hyperparameters](https://en.wikipedia.org/wiki/Hyperparameter_optimization) that need to be set before the training process, such as the number of layers in the network. Because hyperparameters are not part of the model parameters, they cannot be trained using the same loss function. Thus we will try several sets of hyperparameters to train several models and cross-validate them on the validation set to pick the best one; finally, the selected trained model is tested on the test set. Because our model is relatively simple, we will omit this validation process.\n", + "\n", + "\n", + "## Training\n", + "\n", + "`fit_a_line/trainer.py` demonstrates the training using [PaddlePaddle](http://paddlepaddle.org).\n", + "\n", + "### Initialize PaddlePaddle\n", + "\n" + ] + }, + { + "cell_type": "code", + "metadata": { + "editable": true + }, + "source": [ + "paddle.init(use_gpu=False, trainer_count=1)\n" + ], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n" + ] + } + ], + "execution_count": 1 + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "### Model Configuration\n", + "\n", + "Logistic regression is essentially a fully-connected layer with linear activation:\n", + "\n" + ] + }, + { + "cell_type": "code", + "metadata": { + "editable": true + }, + "source": [ + "x = paddle.layer.data(name='x', type=paddle.data_type.dense_vector(13))\n", + "y_predict = paddle.layer.fc(input=x,\n", + " size=1,\n", + " act=paddle.activation.Linear())\n", + "y = paddle.layer.data(name='y', type=paddle.data_type.dense_vector(1))\n", + "cost = paddle.layer.regression_cost(input=y_predict, label=y)\n" + ], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n" + ] + } + ], + "execution_count": 1 + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Create Parameters\n", + "\n" + ] + }, + { + "cell_type": "code", + "metadata": { + "editable": true + }, + "source": [ + "parameters = paddle.parameters.create(cost)\n" + ], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n" + ] + } + ], + "execution_count": 1 + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "### Create Trainer\n", + "\n" + ] + }, + { + "cell_type": "code", + "metadata": { + "editable": true + }, + "source": [ + "optimizer = paddle.optimizer.Momentum(momentum=0)\n", + "\n", + "trainer = paddle.trainer.SGD(cost=cost,\n", + " parameters=parameters,\n", + " update_equation=optimizer)\n" + ], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n" + ] + } + ], + "execution_count": 1 + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "### Feeding Data\n", + "\n", + "PaddlePaddle provides the\n", + "[reader mechanism](https://github.com/PaddlePaddle/Paddle/tree/develop/doc/design/reader)\n", + "for loadinng training data. A reader may return multiple columns, and we need a Python dictionary to specify the mapping from column index to data layers.\n", + "\n" + ] + }, + { + "cell_type": "code", + "metadata": { + "editable": true + }, + "source": [ + "feeding={'x': 0, 'y': 1}\n" + ], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n" + ] + } + ], + "execution_count": 1 + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "Moreover, an event handler is provided to print the training progress:\n", + "\n" + ] + }, + { + "cell_type": "code", + "metadata": { + "editable": true + }, + "source": [ + "# event_handler to print training and testing info\n", + "def event_handler(event):\n", + " if isinstance(event, paddle.event.EndIteration):\n", + " if event.batch_id % 100 == 0:\n", + " print \"Pass %d, Batch %d, Cost %f\" % (\n", + " event.pass_id, event.batch_id, event.cost)\n", + "\n", + " if isinstance(event, paddle.event.EndPass):\n", + " result = trainer.test(\n", + " reader=paddle.batch(\n", + " uci_housing.test(), batch_size=2),\n", + " feeding=feeding)\n", + " print \"Test %d, Cost %f\" % (event.pass_id, result.cost)\n" + ], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n" + ] + } + ], + "execution_count": 1 + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "### Start Training\n", + "\n" + ] + }, + { + "cell_type": "code", + "metadata": { + "editable": true + }, + "source": [ + "trainer.train(\n", + " reader=paddle.batch(\n", + " paddle.reader.shuffle(\n", + " uci_housing.train(), buf_size=500),\n", + " batch_size=2),\n", + " feeding=feeding,\n", + " event_handler=event_handler,\n", + " num_passes=30)\n" + ], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n" + ] + } + ], + "execution_count": 1 + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "## Summary\n", + "This chapter introduces *Linear Regression* and how to train and test this model with PaddlePaddle, using the UCI Housing Data Set. Because a large number of more complex models and techniques are derived from linear regression, it is important to understand its underlying theory and limitation.\n", + "\n", + "\n", + "## References\n", + "1. https://en.wikipedia.org/wiki/Linear_regression\n", + "2. Friedman J, Hastie T, Tibshirani R. The elements of statistical learning[M]. Springer, Berlin: Springer series in statistics, 2001.\n", + "3. Murphy K P. Machine learning: a probabilistic perspective[M]. MIT press, 2012.\n", + "4. Bishop C M. Pattern recognition[J]. Machine Learning, 2006, 128.\n", + "\n", + "\u003cbr/\u003e\n", + "This tutorial is contributed by \u003ca xmlns:cc=\"http://creativecommons.org/ns#\" href=\"http://book.paddlepaddle.org\" property=\"cc:attributionName\" rel=\"cc:attributionURL\"\u003ePaddlePaddle\u003c/a\u003e, and licensed under a \u003ca rel=\"license\" href=\"http://creativecommons.org/licenses/by-nc-sa/4.0/\"\u003eCreative Commons Attribution-NonCommercial-ShareAlike 4.0 International License\u003c/a\u003e.\n" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.6.0" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +} diff --git a/fit_a_line/README.en.md b/fit_a_line/README.en.md index 29aabdab045feeea895b5c3c1e3bd2840f170310..eeef844d24c4ceb25028d7ec85a5335f82d5c51d 100644 --- a/fit_a_line/README.en.md +++ b/fit_a_line/README.en.md @@ -1,7 +1,7 @@ # Linear Regression Let us begin the tutorial with a classical problem called Linear Regression \[[1](#References)\]. In this chapter, we will train a model from a realistic dataset to predict home prices. Some important concepts in Machine Learning will be covered through this example. -The source code for this tutorial lives on [book/fit_a_line](https://github.com/PaddlePaddle/book/tree/develop/fit_a_line). For instructions on getting started with PaddlePaddle, see [PaddlePaddle installation guide](http://www.paddlepaddle.org/doc_cn/build_and_install/index.html). +The source code for this tutorial lives on [book/fit_a_line](https://github.com/PaddlePaddle/book/tree/develop/fit_a_line). For instructions on getting started with PaddlePaddle, see [PaddlePaddle installation guide](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/getstarted/build_and_install/docker_install_en.rst). ## Problem Setup Suppose we have a dataset of $n$ real estate properties. These real estate properties will be referred to as *homes* in this chapter for clarity. @@ -202,4 +202,4 @@ This chapter introduces *Linear Regression* and how to train and test this model 4. Bishop C M. Pattern recognition[J]. Machine Learning, 2006, 128.
-Common Creative License This tutorial was created and published with [Creative Common License 4.0](http://creativecommons.org/licenses/by-nc-sa/4.0/). +This tutorial is contributed by PaddlePaddle, and licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. diff --git a/fit_a_line/README.ipynb b/fit_a_line/README.ipynb new file mode 100644 index 0000000000000000000000000000000000000000..c690570a8a4ef048f2e3460a4b095919bf041f60 --- /dev/null +++ b/fit_a_line/README.ipynb @@ -0,0 +1,407 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# 线性回归\n", + "让我们从经典的线性回归（Linear Regression \\[[1](#参考文献)\\]）模型开始这份教程。在这一章里，你将使用真实的数据集建立起一个房价预测模型，并且了解到机器学习中的若干重要概念。\n", + "\n", + "本教程源代码目录在[book/fit_a_line](https://github.com/PaddlePaddle/book/tree/develop/fit_a_line)， 初次使用请参考PaddlePaddle[安装教程](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/getstarted/build_and_install/docker_install_cn.rst)。\n", + "\n", + "## 背景介绍\n", + "给定一个大小为$n$的数据集 ${\\{y_{i}, x_{i1}, ..., x_{id}\\}}_{i=1}^{n}$，其中$x_{i1}, \\ldots, x_{id}$是第$i$个样本$d$个属性上的取值，$y_i$是该样本待预测的目标。线性回归模型假设目标$y_i$可以被属性间的线性组合描述，即\n", + "\n", + "$$y_i = \\omega_1x_{i1} + \\omega_2x_{i2} + \\ldots + \\omega_dx_{id} + b, i=1,\\ldots,n$$\n", + "\n", + "例如，在我们将要建模的房价预测问题里，$x_{ij}$是描述房子$i$的各种属性（比如房间的个数、周围学校和医院的个数、交通状况等），而 $y_i$是房屋的价格。\n", + "\n", + "初看起来，这个假设实在过于简单了，变量间的真实关系很难是线性的。但由于线性回归模型有形式简单和易于建模分析的优点，它在实际问题中得到了大量的应用。很多经典的统计学习、机器学习书籍\\[[2,3,4](#参考文献)\\]也选择对线性模型独立成章重点讲解。\n", + "\n", + "## 效果展示\n", + "我们使用从[UCI Housing Data Set](https://archive.ics.uci.edu/ml/datasets/Housing)获得的波士顿房价数据集进行模型的训练和预测。下面的散点图展示了使用模型对部分房屋价格进行的预测。其中，每个点的横坐标表示同一类房屋真实价格的中位数，纵坐标表示线性回归模型根据特征预测的结果，当二者值完全相等的时候就会落在虚线上。所以模型预测得越准确，则点离虚线越近。\n", + "\u003cp align=\"center\"\u003e\n", + " \u003cimg src = \"image/predictions.png\" width=400\u003e\u003cbr/\u003e\n", + " 图1. 预测值 V.S. 真实值\n", + "\u003c/p\u003e\n", + "\n", + "## 模型概览\n", + "\n", + "### 模型定义\n", + "\n", + "在波士顿房价数据集中，和房屋相关的值共有14个：前13个用来描述房屋相关的各种信息，即模型中的 $x_i$；最后一个值为我们要预测的该类房屋价格的中位数，即模型中的 $y_i$。因此，我们的模型就可以表示成：\n", + "\n", + "$$\\hat{Y} = \\omega_1X_{1} + \\omega_2X_{2} + \\ldots + \\omega_{13}X_{13} + b$$\n", + "\n", + "$\\hat{Y}$ 表示模型的预测结果，用来和真实值$Y$区分。模型要学习的参数即：$\\omega_1, \\ldots, \\omega_{13}, b$。\n", + "\n", + "建立模型后，我们需要给模型一个优化目标，使得学到的参数能够让预测值$\\hat{Y}$尽可能地接近真实值$Y$。这里我们引入损失函数（[Loss Function](https://en.wikipedia.org/wiki/Loss_function)，或Cost Function）这个概念。 输入任意一个数据样本的目标值$y_{i}$和模型给出的预测值$\\hat{y_{i}}$，损失函数输出一个非负的实值。这个实质通常用来反映模型误差的大小。\n", + "\n", + "对于线性回归模型来讲，最常见的损失函数就是均方误差（Mean Squared Error， [MSE](https://en.wikipedia.org/wiki/Mean_squared_error)）了，它的形式是：\n", + "\n", + "$$MSE=\\frac{1}{n}\\sum_{i=1}^{n}{(\\hat{Y_i}-Y_i)}^2$$\n", + "\n", + "即对于一个大小为$n$的测试集，$MSE$是$n$个数据预测结果误差平方的均值。\n", + "\n", + "### 训练过程\n", + "\n", + "定义好模型结构之后，我们要通过以下几个步骤进行模型训练\n", + " 1. 初始化参数，其中包括权重$\\omega_i$和偏置$b$，对其进行初始化（如0均值，1方差）。\n", + " 2. 网络正向传播计算网络输出和损失函数。\n", + " 3. 根据损失函数进行反向误差传播 （[backpropagation](https://en.wikipedia.org/wiki/Backpropagation)），将网络误差从输出层依次向前传递, 并更新网络中的参数。\n", + " 4. 重复2~3步骤，直至网络训练误差达到规定的程度或训练轮次达到设定值。\n", + "\n", + "## 数据集\n", + "\n", + "### 数据集接口的封装\n", + "首先加载需要的包\n", + "\n" + ] + }, + { + "cell_type": "code", + "metadata": { + "editable": true + }, + "source": [ + "import paddle.v2 as paddle\n", + "import paddle.v2.dataset.uci_housing as uci_housing\n" + ], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n" + ] + } + ], + "execution_count": 1 + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "我们通过uci_housing模块引入了数据集合[UCI Housing Data Set](https://archive.ics.uci.edu/ml/datasets/Housing)\n", + "\n", + "其中，在uci_housing模块中封装了：\n", + "\n", + "1. 数据下载的过程。下载数据保存在~/.cache/paddle/dataset/uci_housing/housing.data。\n", + "2. [数据预处理](#数据预处理)的过程。\n", + "\n", + "\n", + "### 数据集介绍\n", + "这份数据集共506行，每行包含了波士顿郊区的一类房屋的相关信息及该类房屋价格的中位数。其各维属性的意义如下：\n", + "\n", + "| 属性名 | 解释 | 类型 |\n", + "| ------| ------ | ------ |\n", + "| CRIM | 该镇的人均犯罪率 | 连续值 |\n", + "| ZN | 占地面积超过25,000平方呎的住宅用地比例 | 连续值 |\n", + "| INDUS | 非零售商业用地比例 | 连续值 |\n", + "| CHAS | 是否邻近 Charles River | 离散值，1=邻近；0=不邻近 |\n", + "| NOX | 一氧化氮浓度 | 连续值 |\n", + "| RM | 每栋房屋的平均客房数 | 连续值 |\n", + "| AGE | 1940年之前建成的自用单位比例 | 连续值 |\n", + "| DIS | 到波士顿5个就业中心的加权距离 | 连续值 |\n", + "| RAD | 到径向公路的可达性指数 | 连续值 |\n", + "| TAX | 全值财产税率 | 连续值 |\n", + "| PTRATIO | 学生与教师的比例 | 连续值 |\n", + "| B | 1000(BK - 0.63)^2，其中BK为黑人占比 | 连续值 |\n", + "| LSTAT | 低收入人群占比 | 连续值 |\n", + "| MEDV | 同类房屋价格的中位数 | 连续值 |\n", + "\n", + "### 数据预处理\n", + "#### 连续值与离散值\n", + "观察一下数据，我们的第一个发现是：所有的13维属性中，有12维的连续值和1维的离散值（CHAS）。离散值虽然也常使用类似0、1、2这样的数字表示，但是其含义与连续值是不同的，因为这里的差值没有实际意义。例如，我们用0、1、2来分别表示红色、绿色和蓝色的话，我们并不能因此说“蓝色和红色”比“绿色和红色”的距离更远。所以通常对一个有$d$个可能取值的离散属性，我们会将它们转为$d$个取值为0或1的二值属性或者将每个可能取值映射为一个多维向量。不过就这里而言，因为CHAS本身就是一个二值属性，就省去了这个麻烦。\n", + "\n", + "#### 属性的归一化\n", + "另外一个稍加观察即可发现的事实是，各维属性的取值范围差别很大（如图2所示）。例如，属性B的取值范围是[0.32, 396.90]，而属性NOX的取值范围是[0.3850, 0.8170]。这里就要用到一个常见的操作-归一化（normalization）了。归一化的目标是把各位属性的取值范围放缩到差不多的区间，例如[-0.5,0.5]。这里我们使用一种很常见的操作方法：减掉均值，然后除以原取值范围。\n", + "\n", + "做归一化（或 [Feature scaling](https://en.wikipedia.org/wiki/Feature_scaling)）至少有以下3个理由：\n", + "- 过大或过小的数值范围会导致计算时的浮点上溢或下溢。\n", + "- 不同的数值范围会导致不同属性对模型的重要性不同（至少在训练的初始阶段如此），而这个隐含的假设常常是不合理的。这会对优化的过程造成困难，使训练时间大大的加长。\n", + "- 很多的机器学习技巧/模型（例如L1，L2正则项，向量空间模型-Vector Space Model）都基于这样的假设：所有的属性取值都差不多是以0为均值且取值范围相近的。\n", + "\n", + "\u003cp align=\"center\"\u003e\n", + " \u003cimg src = \"image/ranges.png\" width=550\u003e\u003cbr/\u003e\n", + " 图2. 各维属性的取值范围\n", + "\u003c/p\u003e\n", + "\n", + "#### 整理训练集与测试集\n", + "我们将数据集分割为两份：一份用于调整模型的参数，即进行模型的训练，模型在这份数据集上的误差被称为**训练误差**；另外一份被用来测试，模型在这份数据集上的误差被称为**测试误差**。我们训练模型的目的是为了通过从训练数据中找到规律来预测未知的新数据，所以测试误差是更能反映模型表现的指标。分割数据的比例要考虑到两个因素：更多的训练数据会降低参数估计的方差，从而得到更可信的模型；而更多的测试数据会降低测试误差的方差，从而得到更可信的测试误差。我们这个例子中设置的分割比例为$8:2$\n", + "\n", + "\n", + "在更复杂的模型训练过程中，我们往往还会多使用一种数据集：验证集。因为复杂的模型中常常还有一些超参数（[Hyperparameter](https://en.wikipedia.org/wiki/Hyperparameter_optimization)）需要调节，所以我们会尝试多种超参数的组合来分别训练多个模型，然后对比它们在验证集上的表现选择相对最好的一组超参数，最后才使用这组参数下训练的模型在测试集上评估测试误差。由于本章训练的模型比较简单，我们暂且忽略掉这个过程。\n", + "\n", + "## 训练\n", + "\n", + "`fit_a_line/trainer.py`演示了训练的整体过程。\n", + "\n", + "### 初始化PaddlePaddle\n", + "\n" + ] + }, + { + "cell_type": "code", + "metadata": { + "editable": true + }, + "source": [ + "paddle.init(use_gpu=False, trainer_count=1)\n" + ], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n" + ] + } + ], + "execution_count": 1 + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "### 模型配置\n", + "\n", + "线性回归的模型其实就是一个采用线性激活函数（linear activation，`LinearActivation`）的全连接层（fully-connected layer，`fc_layer`）：\n", + "\n" + ] + }, + { + "cell_type": "code", + "metadata": { + "editable": true + }, + "source": [ + "x = paddle.layer.data(name='x', type=paddle.data_type.dense_vector(13))\n", + "y_predict = paddle.layer.fc(input=x,\n", + " size=1,\n", + " act=paddle.activation.Linear())\n", + "y = paddle.layer.data(name='y', type=paddle.data_type.dense_vector(1))\n", + "cost = paddle.layer.regression_cost(input=y_predict, label=y)\n" + ], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n" + ] + } + ], + "execution_count": 1 + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 创建参数\n", + "\n" + ] + }, + { + "cell_type": "code", + "metadata": { + "editable": true + }, + "source": [ + "parameters = paddle.parameters.create(cost)\n" + ], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n" + ] + } + ], + "execution_count": 1 + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "### 创建Trainer\n", + "\n" + ] + }, + { + "cell_type": "code", + "metadata": { + "editable": true + }, + "source": [ + "optimizer = paddle.optimizer.Momentum(momentum=0)\n", + "\n", + "trainer = paddle.trainer.SGD(cost=cost,\n", + " parameters=parameters,\n", + " update_equation=optimizer)\n" + ], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n" + ] + } + ], + "execution_count": 1 + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "### 读取数据且打印训练的中间信息\n", + "\n", + "PaddlePaddle提供一个\n", + "[reader机制](https://github.com/PaddlePaddle/Paddle/tree/develop/doc/design/reader)\n", + "来读取数据。 Reader返回的数据可以包括多列，我们需要一个Python dict把列\n", + "序号映射到网络里的数据层。\n", + "\n" + ] + }, + { + "cell_type": "code", + "metadata": { + "editable": true + }, + "source": [ + "feeding={'x': 0, 'y': 1}\n" + ], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n" + ] + } + ], + "execution_count": 1 + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "此外，我们还可以提供一个 event handler，来打印训练的进度：\n", + "\n" + ] + }, + { + "cell_type": "code", + "metadata": { + "editable": true + }, + "source": [ + "# event_handler to print training and testing info\n", + "def event_handler(event):\n", + " if isinstance(event, paddle.event.EndIteration):\n", + " if event.batch_id % 100 == 0:\n", + " print \"Pass %d, Batch %d, Cost %f\" % (\n", + " event.pass_id, event.batch_id, event.cost)\n", + "\n", + " if isinstance(event, paddle.event.EndPass):\n", + " result = trainer.test(\n", + " reader=paddle.batch(\n", + " uci_housing.test(), batch_size=2),\n", + " feeding=feeding)\n", + " print \"Test %d, Cost %f\" % (event.pass_id, result.cost)\n" + ], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n" + ] + } + ], + "execution_count": 1 + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "### 开始训练\n", + "\n" + ] + }, + { + "cell_type": "code", + "metadata": { + "editable": true + }, + "source": [ + "trainer.train(\n", + " reader=paddle.batch(\n", + " paddle.reader.shuffle(\n", + " uci_housing.train(), buf_size=500),\n", + " batch_size=2),\n", + " feeding=feeding,\n", + " event_handler=event_handler,\n", + " num_passes=30)\n" + ], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n" + ] + } + ], + "execution_count": 1 + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "## 总结\n", + "在这章里，我们借助波士顿房价这一数据集，介绍了线性回归模型的基本概念，以及如何使用PaddlePaddle实现训练和测试的过程。很多的模型和技巧都是从简单的线性回归模型演化而来，因此弄清楚线性模型的原理和局限非常重要。\n", + "\n", + "\n", + "## 参考文献\n", + "1. https://en.wikipedia.org/wiki/Linear_regression\n", + "2. Friedman J, Hastie T, Tibshirani R. The elements of statistical learning[M]. Springer, Berlin: Springer series in statistics, 2001.\n", + "3. Murphy K P. Machine learning: a probabilistic perspective[M]. MIT press, 2012.\n", + "4. Bishop C M. Pattern recognition[J]. Machine Learning, 2006, 128.\n", + "\n", + "\u003cbr/\u003e\n", + "\u003ca rel=\"license\" href=\"http://creativecommons.org/licenses/by-nc-sa/4.0/\"\u003e\u003cimg alt=\"知识共享许可协议\" style=\"border-width:0\" src=\"https://i.creativecommons.org/l/by-nc-sa/4.0/88x31.png\" /\u003e\u003c/a\u003e\u003cbr /\u003e\u003cspan xmlns:dct=\"http://purl.org/dc/terms/\" href=\"http://purl.org/dc/dcmitype/Text\" property=\"dct:title\" rel=\"dct:type\"\u003e本教程\u003c/span\u003e 由 \u003ca xmlns:cc=\"http://creativecommons.org/ns#\" href=\"http://book.paddlepaddle.org\" property=\"cc:attributionName\" rel=\"cc:attributionURL\"\u003ePaddlePaddle\u003c/a\u003e 创作，采用 \u003ca rel=\"license\" href=\"http://creativecommons.org/licenses/by-nc-sa/4.0/\"\u003e知识共享 署名-非商业性使用-相同方式共享 4.0 国际 许可协议\u003c/a\u003e进行许可。\n" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.6.0" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +} diff --git a/fit_a_line/README.md b/fit_a_line/README.md index 266c6e91cb4d5249997203cada7ef920d3744386..6ae43bcc591e51b9ff91c61f2c8ae7e6ff407c25 100644 --- a/fit_a_line/README.md +++ b/fit_a_line/README.md @@ -1,7 +1,7 @@ # 线性回归 让我们从经典的线性回归（Linear Regression \[[1](#参考文献)\]）模型开始这份教程。在这一章里，你将使用真实的数据集建立起一个房价预测模型，并且了解到机器学习中的若干重要概念。 -本教程源代码目录在[book/fit_a_line](https://github.com/PaddlePaddle/book/tree/develop/fit_a_line)， 初次使用请参考PaddlePaddle[安装教程](http://www.paddlepaddle.org/doc_cn/build_and_install/index.html)。 +本教程源代码目录在[book/fit_a_line](https://github.com/PaddlePaddle/book/tree/develop/fit_a_line)， 初次使用请参考PaddlePaddle[安装教程](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/getstarted/build_and_install/docker_install_cn.rst)。 ## 背景介绍 给定一个大小为$n$的数据集 ${\{y_{i}, x_{i1}, ..., x_{id}\}}_{i=1}^{n}$，其中$x_{i1}, \ldots, x_{id}$是第$i$个样本$d$个属性上的取值，$y_i$是该样本待预测的目标。线性回归模型假设目标$y_i$可以被属性间的线性组合描述，即 @@ -15,8 +15,8 @@ $$y_i = \omega_1x_{i1} + \omega_2x_{i2} + \ldots + \omega_dx_{id} + b, i=1,\ldo ## 效果展示 我们使用从[UCI Housing Data Set](https://archive.ics.uci.edu/ml/datasets/Housing)获得的波士顿房价数据集进行模型的训练和预测。下面的散点图展示了使用模型对部分房屋价格进行的预测。其中，每个点的横坐标表示同一类房屋真实价格的中位数，纵坐标表示线性回归模型根据特征预测的结果，当二者值完全相等的时候就会落在虚线上。所以模型预测得越准确，则点离虚线越近。

-
- 图1. 预测值 V.S. 真实值 +
+ 图1. 预测值 V.S. 真实值

## 模型概览 @@ -96,8 +96,8 @@ import paddle.v2.dataset.uci_housing as uci_housing - 很多的机器学习技巧/模型（例如L1，L2正则项，向量空间模型-Vector Space Model）都基于这样的假设：所有的属性取值都差不多是以0为均值且取值范围相近的。

-
- 图2. 各维属性的取值范围 +
+ 图2. 各维属性的取值范围

#### 整理训练集与测试集 diff --git a/fit_a_line/index.en.html b/fit_a_line/index.en.html index 221fffdbf142ea7f0892aa34a857cde1ae173d08..22d6aeaa7da7ef9277ca59ac29f8b0e62ba78fa3 100644 --- a/fit_a_line/index.en.html +++ b/fit_a_line/index.en.html @@ -43,7 +43,7 @@ # Linear Regression Let us begin the tutorial with a classical problem called Linear Regression \[[1](#References)\]. In this chapter, we will train a model from a realistic dataset to predict home prices. Some important concepts in Machine Learning will be covered through this example. -The source code for this tutorial lives on [book/fit_a_line](https://github.com/PaddlePaddle/book/tree/develop/fit_a_line). For instructions on getting started with PaddlePaddle, see [PaddlePaddle installation guide](http://www.paddlepaddle.org/doc_cn/build_and_install/index.html). +The source code for this tutorial lives on [book/fit_a_line](https://github.com/PaddlePaddle/book/tree/develop/fit_a_line). For instructions on getting started with PaddlePaddle, see [PaddlePaddle installation guide](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/getstarted/build_and_install/docker_install_en.rst). ## Problem Setup Suppose we have a dataset of $n$ real estate properties. These real estate properties will be referred to as *homes* in this chapter for clarity. @@ -244,7 +244,7 @@ This chapter introduces *Linear Regression* and how to train and test this model 4. Bishop C M. Pattern recognition[J]. Machine Learning, 2006, 128.
-Common Creative License This tutorial was created and published with [Creative Common License 4.0](http://creativecommons.org/licenses/by-nc-sa/4.0/). +This tutorial is contributed by PaddlePaddle, and licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. diff --git a/fit_a_line/index.html b/fit_a_line/index.html index 7c1b57ab6805f2a1df274cd9cc55cceebefc30c5..495d8e39726651c4efb2ce96d3c5da265bcfb171 100644 --- a/fit_a_line/index.html +++ b/fit_a_line/index.html @@ -43,7 +43,7 @@ # 线性回归 让我们从经典的线性回归（Linear Regression \[[1](#参考文献)\]）模型开始这份教程。在这一章里，你将使用真实的数据集建立起一个房价预测模型，并且了解到机器学习中的若干重要概念。 -本教程源代码目录在[book/fit_a_line](https://github.com/PaddlePaddle/book/tree/develop/fit_a_line)， 初次使用请参考PaddlePaddle[安装教程](http://www.paddlepaddle.org/doc_cn/build_and_install/index.html)。 +本教程源代码目录在[book/fit_a_line](https://github.com/PaddlePaddle/book/tree/develop/fit_a_line)， 初次使用请参考PaddlePaddle[安装教程](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/getstarted/build_and_install/docker_install_cn.rst)。 ## 背景介绍 给定一个大小为$n$的数据集 ${\{y_{i}, x_{i1}, ..., x_{id}\}}_{i=1}^{n}$，其中$x_{i1}, \ldots, x_{id}$是第$i$个样本$d$个属性上的取值，$y_i$是该样本待预测的目标。线性回归模型假设目标$y_i$可以被属性间的线性组合描述，即 @@ -57,8 +57,8 @@ $$y_i = \omega_1x_{i1} + \omega_2x_{i2} + \ldots + \omega_dx_{id} + b, i=1,\ldo ## 效果展示 我们使用从[UCI Housing Data Set](https://archive.ics.uci.edu/ml/datasets/Housing)获得的波士顿房价数据集进行模型的训练和预测。下面的散点图展示了使用模型对部分房屋价格进行的预测。其中，每个点的横坐标表示同一类房屋真实价格的中位数，纵坐标表示线性回归模型根据特征预测的结果，当二者值完全相等的时候就会落在虚线上。所以模型预测得越准确，则点离虚线越近。

-
- 图1. 预测值 V.S. 真实值 +
+ 图1. 预测值 V.S. 真实值

## 模型概览 @@ -138,8 +138,8 @@ import paddle.v2.dataset.uci_housing as uci_housing - 很多的机器学习技巧/模型（例如L1，L2正则项，向量空间模型-Vector Space Model）都基于这样的假设：所有的属性取值都差不多是以0为均值且取值范围相近的。

-
- 图2. 各维属性的取值范围 +
+ 图2. 各维属性的取值范围

#### 整理训练集与测试集 diff --git a/fit_a_line/train.py b/fit_a_line/train.py index df665c436872bbaaf5c08790cba69c8ac17e5db7..6fae9e012e6153c6fc84a30ea72d82f2d9a80200 100644 --- a/fit_a_line/train.py +++ b/fit_a_line/train.py @@ -18,9 +18,8 @@ def main(): # create optimizer optimizer = paddle.optimizer.Momentum(momentum=0) - trainer = paddle.trainer.SGD(cost=cost, - parameters=parameters, - update_equation=optimizer) + trainer = paddle.trainer.SGD( + cost=cost, parameters=parameters, update_equation=optimizer) feeding = {'x': 0, 'y': 1} @@ -33,16 +32,14 @@ def main(): if isinstance(event, paddle.event.EndPass): result = trainer.test( - reader=paddle.batch( - uci_housing.test(), batch_size=2), + reader=paddle.batch(uci_housing.test(), batch_size=2), feeding=feeding) print "Test %d, Cost %f" % (event.pass_id, result.cost) # training trainer.train( reader=paddle.batch( - paddle.reader.shuffle( - uci_housing.train(), buf_size=500), + paddle.reader.shuffle(uci_housing.train(), buf_size=500), batch_size=2), feeding=feeding, event_handler=event_handler, diff --git a/image_classification/README.en.md b/image_classification/README.en.md index 3cfc40cbe339d274cf055d62ba1b7c9d9e7955a5..57aac0935b94faf11a33c3291d0c6991075cd698 100644 --- a/image_classification/README.en.md +++ b/image_classification/README.en.md @@ -1,7 +1,7 @@ Image Classification ======================= -The source code of this chapter is in [book/image_classification](https://github.com/PaddlePaddle/book/tree/develop/image_classification). For the first-time users, please refer to PaddlePaddle[Installation Tutorial](http://www.paddlepaddle.org/doc_cn/build_and_install/index.html) for installation instructions. +The source code of this chapter is in [book/image_classification](https://github.com/PaddlePaddle/book/tree/develop/image_classification). For the first-time users, please refer to PaddlePaddle [Installation Tutorial](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/getstarted/build_and_install/docker_install_en.rst) for installation instructions. ## Background @@ -135,146 +135,73 @@ Figure 10. ResNet model for ImageNet

-## Data Preparation - -### Data description and downloading +## Dataset Commonly used public datasets for image classification are CIFAR(https://www.cs.toronto.edu/~kriz/cifar.html), ImageNet(http://image-net.org/), COCO(http://mscoco.org/), etc. Those used for fine-grained image classification are CUB-200-2011(http://www.vision.caltech.edu/visipedia/CUB-200-2011.html), Stanford Dog(http://vision.stanford.edu/aditya86/ImageNetDogs/), Oxford-flowers(http://www.robots.ox.ac.uk/~vgg/data/flowers/), etc. Among them, ImageNet are the largest and most research results are reported on ImageNet as mentioned in Model Overview section. Since 2010, the data of Imagenet has gone through some changes. The commonly used ImageNet-2012 dataset contains 1000 categories. There are 1,281,167 training images, ranging from 732 to 1200 images per category, and 50,000 validation images with 50 images per category in average. -Since ImageNet is too large to be downloaded and trained efficiently, we use CIFAR10 (https://www.cs.toronto.edu/~kriz/cifar.html) in this tutorial. The CIFAR-10 dataset consists of 60000 32x32 color images in 10 classes, with 6000 images per class. There are 50000 training images and 10000 test images. Figure 11 shows all the classes in CIFAR10 as well as 10 images randomly sampled from each category. +Since ImageNet is too large to be downloaded and trained efficiently, we use CIFAR-10 (https://www.cs.toronto.edu/~kriz/cifar.html) in this tutorial. The CIFAR-10 dataset consists of 60000 32x32 color images in 10 classes, with 6000 images per class. There are 50000 training images and 10000 test images. Figure 11 shows all the classes in CIFAR-10 as well as 10 images randomly sampled from each category.


Figure 11. CIFAR10 dataset[21]

-The following command is used for downloading data and calculating the mean image used for data preprocessing. - -```bash -./data/get_data.sh -``` + `paddle.datasets` package encapsulates multiple public datasets, including `cifar`, `imdb`, `mnist`, `moivelens` and `wmt14`, etc. There's no need for us to manually download and preprocess CIFAR-10. -### Data provider for PaddlePaddle +After issuing a command `python train.py`, training will starting immediately. The details will be unpacked by the following sessions to see how it works. -We use Python interface for providing data to PaddlePaddle. The following file dataprovider.py is a complete example for CIFAR10. +## Model Structure -- 'initializer' function performs initialization of dataprovider: loading the mean image, defining two input types -- image and label. +### Initialize PaddlePaddle -- 'process' function sends preprocessed data to PaddlePaddle. Data preprocessing performed in this function includes data perturbation, random horizontal flipping, deducting mean image from the raw image. +We must import and initialize PaddlePaddle (enable/disable GPU, set the number of trainers, etc). ```python -import numpy as np -import cPickle -from paddle.trainer.PyDataProvider2 import * - -def initializer(settings, mean_path, is_train, **kwargs): - settings.is_train = is_train - settings.input_size = 3 * 32 * 32 - settings.mean = np.load(mean_path)['mean'] - settings.input_types = { - 'image': dense_vector(settings.input_size), - 'label': integer_value(10) - } - - -@provider(init_hook=initializer, pool_size=50000) -def process(settings, file_list): - with open(file_list, 'r') as fdata: - for fname in fdata: - fo = open(fname.strip(), 'rb') - batch = cPickle.load(fo) - fo.close() - images = batch['data'] - labels = batch['labels'] - for im, lab in zip(images, labels): - if settings.is_train and np.random.randint(2): - im = im.reshape(3, 32, 32) - im = im[:,:,::-1] - im = im.flatten() - im = im - settings.mean - yield { - 'image': im.astype('float32'), - 'label': int(lab) - } -``` +import sys +import paddle.v2 as paddle -## Model Config - -### Data Definition - -In model config file, function `define_py_data_sources2` sets argument 'module' to dataprovider file for loading data, 'args' to mean image file. If the config file is used for prediction, then there is no need to set argument 'train_list'. - -```python -from paddle.trainer_config_helpers import * - -is_predict = get_config_arg("is_predict", bool, False) -if not is_predict: - define_py_data_sources2( - train_list='data/train.list', - test_list='data/test.list', - module='dataprovider', - obj='process', - args={'mean_path': 'data/mean.meta'}) -``` - -### Algorithm Settings - -In model config file, function 'settings' specifies optimization algorithm, batch size, learning rate, momentum and L2 regularization. - -```python -settings( - batch_size=128, - learning_rate=0.1 / 128.0, - learning_rate_decay_a=0.1, - learning_rate_decay_b=50000 * 100, - learning_rate_schedule='discexp', - learning_method=MomentumOptimizer(0.9), - regularization=L2Regularization(0.0005 * 128),) +# PaddlePaddle init +paddle.init(use_gpu=False, trainer_count=1) ``` -The learning rate adjustment policy can be defined with variables `learning_rate_decay_a`($a$), `learning_rate_decay_b`($b$) and `learning_rate_schedule`. In this example, discrete exponential method is used for adjusting learning rate. The formula is as follows, -$$ lr = lr_{0} * a^ {\lfloor \frac{n}{ b}\rfloor} $$ -where $n$ is the number of processed samples, $lr_{0}$ is the learning_rate set in 'settings'. - -### Model Architecture - -Here we provide the cofig files for VGG and ResNet models. +As alluded to in section [Model Overview](#model-overview), here we provide the implementations of both VGG and ResNet models. -#### VGG +### VGG -First we define VGG network. Since the image size and amount of CIFAR10 are relatively small comparing to ImageNet, we uses a small version of VGG network for CIFAR10. Convolution groups incorporate BN and dropout operations. +First, we use a VGG network. Since the image size and amount of CIFAR10 are relatively small comparing to ImageNet, we uses a small version of VGG network for CIFAR10. Convolution groups incorporate BN and dropout operations. 1. Define input data and its dimension - The input to the network is defined as `data_layer`, or image pixels in the context of image classification. The images in CIFAR10 are 32x32 color images of three channels. Therefore, the size of the input data is 3072 (3x32x32), and the number of categories is 10. + The input to the network is defined as `paddle.layer.data`, or image pixels in the context of image classification. The images in CIFAR10 are 32x32 color images of three channels. Therefore, the size of the input data is 3072 (3x32x32), and the number of categories is 10. ```python datadim = 3 * 32 * 32 classdim = 10 - data = data_layer(name='image', size=datadim) + image = paddle.layer.data( + name="image", type=paddle.data_type.dense_vector(datadim)) ``` 2. Define VGG main module ```python - net = vgg_bn_drop(data) + net = vgg_bn_drop(image) ``` - The input to VGG main module is from data layer. `vgg_bn_drop` defines a 16-layer VGG network, with each convolutional layer followed by BN and dropout layers. Here is the definition in detail: + The input to VGG main module is from the data layer. `vgg_bn_drop` defines a 16-layer VGG network, with each convolutional layer followed by BN and dropout layers. Here is the definition in detail: ```python - def vgg_bn_drop(input, num_channels): - def conv_block(ipt, num_filter, groups, dropouts, num_channels_=None): - return img_conv_group( + def vgg_bn_drop(input): + def conv_block(ipt, num_filter, groups, dropouts, num_channels=None): + return paddle.networks.img_conv_group( input=ipt, - num_channels=num_channels_, + num_channels=num_channels, pool_size=2, pool_stride=2, conv_num_filter=[num_filter] * groups, conv_filter_size=3, - conv_act=ReluActivation(), + conv_act=paddle.activation.Relu(), conv_with_batchnorm=True, conv_batchnorm_drop_rate=dropouts, - pool_type=MaxPooling()) + pool_type=paddle.pooling.Max()) conv1 = conv_block(input, 64, 2, [0.3, 0], 3) conv2 = conv_block(conv1, 128, 2, [0.4, 0]) @@ -282,16 +209,17 @@ First we define VGG network. Since the image size and amount of CIFAR10 are rela conv4 = conv_block(conv3, 512, 3, [0.4, 0.4, 0]) conv5 = conv_block(conv4, 512, 3, [0.4, 0.4, 0]) - drop = dropout_layer(input=conv5, dropout_rate=0.5) - fc1 = fc_layer(input=drop, size=512, act=LinearActivation()) - bn = batch_norm_layer( - input=fc1, act=ReluActivation(), layer_attr=ExtraAttr(drop_rate=0.5)) - fc2 = fc_layer(input=bn, size=512, act=LinearActivation()) + drop = paddle.layer.dropout(input=conv5, dropout_rate=0.5) + fc1 = paddle.layer.fc(input=drop, size=512, act=paddle.activation.Linear()) + bn = paddle.layer.batch_norm( + input=fc1, + act=paddle.activation.Relu(), + layer_attr=paddle.attr.Extra(drop_rate=0.5)) + fc2 = paddle.layer.fc(input=bn, size=512, act=paddle.activation.Linear()) return fc2 - ``` - 2.1. First defines a convolution block or conv_block. The default convolution kernel is 3x3, and the default pooling size is 2x2 with stride 2. Dropout specifies the probability in dropout operation. Function `img_conv_group` is defined in `paddle.trainer_config_helpers` consisting of a series of `Conv->BN->ReLu->Dropout` and a `Pooling`. + 2.1. First defines a convolution block or conv_block. The default convolution kernel is 3x3, and the default pooling size is 2x2 with stride 2. Dropout specifies the probability in dropout operation. Function `img_conv_group` is defined in `paddle.networks` consisting of a series of `Conv->BN->ReLu->Dropout` and a `Pooling`. 2.2. Five groups of convolutions. The first two groups perform two convolutions, while the last three groups perform three convolutions. The dropout rate of the last convolution in each group is set to 0, which means there is no dropout for this layer. @@ -309,15 +237,12 @@ First we define VGG network. Since the image size and amount of CIFAR10 are rela 4. Define Loss Function and Outputs - In the context of supervised learning, labels of training images are defined in `data_layer`, too. During training, cross-entropy is used as loss function and as the output of the network; During testing, the outputs are the probabilities calculated in the classifier. + In the context of supervised learning, labels of training images are defined in `paddle.layer.data`, too. During training, cross-entropy is used as loss function and as the output of the network; During testing, the outputs are the probabilities calculated in the classifier. ```python - if not is_predict: - lbl = data_layer(name="label", size=class_num) - cost = classification_cost(input=out, label=lbl) - outputs(cost) - else: - outputs(out) + lbl = paddle.layer.data( + name="label", type=paddle.data_type.integer_value(classdim)) + cost = paddle.layer.classification_cost(input=out, label=lbl) ``` ### ResNet @@ -325,13 +250,13 @@ First we define VGG network. Since the image size and amount of CIFAR10 are rela The first, third and forth steps of a ResNet are the same as a VGG. The second one is the main module. ```python -net = resnet_cifar10(data, depth=56) +net = resnet_cifar10(data, depth=32) ``` Here are some basic functions used in `resnet_cifar10`: - `conv_bn_layer` : convolutional layer followed by BN. - - `shortcut` : the shortcut branch in a residual block. There are two kinds of shortcuts: 1x1 convolution used when the number of channels between input and output are different; direct connection used otherwise. + - `shortcut` : the shortcut branch in a residual block. There are two kinds of shortcuts: 1x1 convolution used when the number of channels between input and output is different; direct connection used otherwise. - `basicblock` : a basic residual module as shown in the left of Figure 9, consisting of two sequential 3x3 convolutions and one "shortcut" branch. - `bottleneck` : a bottleneck module as shown in the right of Figure 9, consisting of a two 1x1 convolutions with one 3x3 convolution in between branch and a "shortcut" branch. @@ -343,47 +268,38 @@ def conv_bn_layer(input, filter_size, stride, padding, - active_type=ReluActivation(), + active_type=paddle.activation.Relu(), ch_in=None): - tmp = img_conv_layer( + tmp = paddle.layer.img_conv( input=input, filter_size=filter_size, num_channels=ch_in, num_filters=ch_out, stride=stride, padding=padding, - act=LinearActivation(), + act=paddle.activation.Linear(), bias_attr=False) - return batch_norm_layer(input=tmp, act=active_type) - + return paddle.layer.batch_norm(input=tmp, act=active_type) def shortcut(ipt, n_in, n_out, stride): if n_in != n_out: - return conv_bn_layer(ipt, n_out, 1, stride, 0, LinearActivation()) + return conv_bn_layer(ipt, n_out, 1, stride, 0, + paddle.activation.Linear()) else: return ipt def basicblock(ipt, ch_out, stride): - ch_in = ipt.num_filters + ch_in = ch_out * 2 tmp = conv_bn_layer(ipt, ch_out, 3, stride, 1) - tmp = conv_bn_layer(tmp, ch_out, 3, 1, 1, LinearActivation()) - short = shortcut(ipt, ch_in, ch_out, stride) - return addto_layer(input=[ipt, short], act=ReluActivation()) - -def bottleneck(ipt, ch_out, stride): - ch_in = ipt.num_filter - tmp = conv_bn_layer(ipt, ch_out, 1, stride, 0) - tmp = conv_bn_layer(tmp, ch_out, 3, 1, 1) - tmp = conv_bn_layer(tmp, ch_out * 4, 1, 1, 0, LinearActivation()) + tmp = conv_bn_layer(tmp, ch_out, 3, 1, 1, paddle.activation.Linear()) short = shortcut(ipt, ch_in, ch_out, stride) - return addto_layer(input=[ipt, short], act=ReluActivation()) + return paddle.layer.addto(input=[tmp, short], act=paddle.activation.Relu()) def layer_warp(block_func, ipt, features, count, stride): tmp = block_func(ipt, features, stride) for i in range(1, count): tmp = block_func(tmp, features, 1) return tmp - ``` The following are the components of `resnet_cifar10`: @@ -395,106 +311,131 @@ The following are the components of `resnet_cifar10`: Note: besides the first convolutional layer and the last fully-connected layer, the total number of layers in three `layer_warp` should be dividable by 6, that is the depth of `resnet_cifar10` should satisfy $(depth - 2) % 6 == 0$. ```python -def resnet_cifar10(ipt, depth=56): +def resnet_cifar10(ipt, depth=32): # depth should be one of 20, 32, 44, 56, 110, 1202 assert (depth - 2) % 6 == 0 n = (depth - 2) / 6 nStages = {16, 64, 128} - conv1 = conv_bn_layer(ipt, - ch_in=3, - ch_out=16, - filter_size=3, - stride=1, - padding=1) + conv1 = conv_bn_layer( + ipt, ch_in=3, ch_out=16, filter_size=3, stride=1, padding=1) res1 = layer_warp(basicblock, conv1, 16, n, 1) res2 = layer_warp(basicblock, res1, 32, n, 2) res3 = layer_warp(basicblock, res2, 64, n, 2) - pool = img_pool_layer(input=res3, - pool_size=8, - stride=1, - pool_type=AvgPooling()) + pool = paddle.layer.img_pool( + input=res3, pool_size=8, stride=1, pool_type=paddle.pooling.Avg()) return pool ``` ## Model Training -We can train the model by running the script train.sh, which specifies config file, device type, number of threads, number of passes, path to the trained models, etc, +### Define Parameters -``` bash -sh train.sh -``` +First, we create the model parameters according to the previous model configuration `cost`. -Here is an example script `train.sh`: - -```bash -#cfg=models/resnet.py -cfg=models/vgg.py -output=output -log=train.log - -paddle train \ - --config=$cfg \ - --use_gpu=true \ - --trainer_count=1 \ - --log_period=100 \ - --num_passes=300 \ - --save_dir=$output \ - 2>&1 | tee $log +```python +# Create parameters +parameters = paddle.parameters.create(cost) ``` -- `--config=$cfg` : specifies config file. The default is `models/vgg.py`. -- `--use_gpu=true` : uses GPU for training. If use CPU，set it to be false. -- `--trainer_count=1` : specifies the number of threads or GPUs. -- `--log_period=100` : specifies the number of batches between two logs. -- `--save_dir=$output` : specifies the path for saving trained models. +### Create Trainer -Here is an example log after training for one pass. The average error rates are 0.79958 on training set and 0.7858 on validation set. +Before jumping into creating a training module, algorithm setting is also necessary. +Here we specified `Momentum` optimization algorithm via `paddle.optimizer`. -```text -TrainerInternal.cpp:165] Batch=300 samples=38400 AvgCost=2.07708 CurrentCost=1.96158 Eval: classification_error_evaluator=0.81151 CurrentEval: classification_error_evaluator=0.789297 -TrainerInternal.cpp:181] Pass=0 Batch=391 samples=50000 AvgCost=2.03348 Eval: classification_error_evaluator=0.79958 -Tester.cpp:115] Test samples=10000 cost=1.99246 Eval: classification_error_evaluator=0.7858 +```python +# Create optimizer +momentum_optimizer = paddle.optimizer.Momentum( + momentum=0.9, + regularization=paddle.optimizer.L2Regularization(rate=0.0002 * 128), + learning_rate=0.1 / 128.0, + learning_rate_decay_a=0.1, + learning_rate_decay_b=50000 * 100, + learning_rate_schedule='discexp', + batch_size=128) + +# Create trainer +trainer = paddle.trainer.SGD(cost=cost, + parameters=parameters, + update_equation=momentum_optimizer) ``` -Figure 12 shows the curve of training error rate, which indicates it converges at Pass 200 with error rate 8.54%. +The learning rate adjustment policy can be defined with variables `learning_rate_decay_a`($a$), `learning_rate_decay_b`($b$) and `learning_rate_schedule`. In this example, discrete exponential method is used for adjusting learning rate. The formula is as follows, +$$ lr = lr_{0} * a^ {\lfloor \frac{n}{ b}\rfloor} $$ +where $n$ is the number of processed samples, $lr_{0}$ is the learning_rate. -

-
-Figure 12. The error rate of VGG model on CIFAR10 -

+### Training -## Model Application +`cifar.train10()` will yield records during each pass, after shuffling, a batch input is generated for training. -After training is done, the model from each pass is saved in `output/pass-%05d`. For example, the model of Pass 300 is saved in `output/pass-00299`. The script `classify.py` can be used to extract features and to classify an image. The default config file of this script is `models/vgg.py`. +```python +reader=paddle.batch( + paddle.reader.shuffle( + paddle.dataset.cifar.train10(), buf_size=50000), + batch_size=128) +``` + +`feeding` is devoted to specifying the correspondence between each yield record and `paddle.layer.data`. For instance, + the first column of data generated by `cifar.train10()` corresponds to image layer's feature. +```python +feeding={'image': 0, + 'label': 1} +``` -### Prediction +Callback function `event_handler` will be called during training when a pre-defined event happens. -We can run the following script to predict the category of an image. The default device is GPU. If to use CPU, set `-c`. -```bash -python classify.py --job=predict --model=output/pass-00299 --data=image/dog.png # -c +```python +# event handler to track training and testing process +def event_handler(event): + if isinstance(event, paddle.event.EndIteration): + if event.batch_id % 100 == 0: + print "\nPass %d, Batch %d, Cost %f, %s" % ( + event.pass_id, event.batch_id, event.cost, event.metrics) + else: + sys.stdout.write('.') + sys.stdout.flush() + if isinstance(event, paddle.event.EndPass): + result = trainer.test( + reader=paddle.batch( + paddle.dataset.cifar.test10(), batch_size=128), + feeding=feeding) + print "\nTest with Pass %d, %s" % (event.pass_id, result.metrics) ``` -Here is the result： +Finally, we can invoke `trainer.train` to start training: -```text -Label of image/dog.png is: 5 +```python +trainer.train( + reader=reader, + num_passes=200, + event_handler=event_handler, + feeding=feeding) ``` -### Feature Extraction +Here is an example log after training for one pass. The average error rates are 0.6875 on the training set and 0.8852 on the validation set. -We can run the following command to extract features from an image. Here `job` should be `extract` and the default layer is the first convolutional layer. Figure 13 shows the 64 feature maps output from the first convolutional layer of the VGG model. - -```bash -python classify.py --job=extract --model=output/pass-00299 --data=image/dog.png # -c +```text +Pass 0, Batch 0, Cost 2.473182, {'classification_error_evaluator': 0.9140625} +................................................................................................... +Pass 0, Batch 100, Cost 1.913076, {'classification_error_evaluator': 0.78125} +................................................................................................... +Pass 0, Batch 200, Cost 1.783041, {'classification_error_evaluator': 0.7421875} +................................................................................................... +Pass 0, Batch 300, Cost 1.668833, {'classification_error_evaluator': 0.6875} +.......................................................................................... +Test with Pass 0, {'classification_error_evaluator': 0.885200023651123} ``` +Figure 12 shows the curve of training error rate, which indicates it converges at Pass 200 with error rate 8.54%.

-
-Figre 13. Visualization of convolution layer feature maps +
+Figure 12. The error rate of VGG model on CIFAR10

+ +After training is done, the model from each pass is saved in `output/pass-%05d`. For example, the model of Pass 300 is saved in `output/pass-00299`. + ## Conclusion Traditional image classification methods involve multiple stages of processing and the framework is very complicated. In contrast, CNN models can be trained end-to-end with significant increase of classification accuracy. In this chapter, we introduce three models -- VGG, GoogleNet, ResNet, provide PaddlePaddle config files for training VGG and ResNet on CIFAR10, and explain how to perform prediction and feature extraction using PaddlePaddle API. For other datasets such as ImageNet, the procedure for config and training are the same and you are welcome to give it a try. @@ -547,4 +488,4 @@ Traditional image classification methods involve multiple stages of processing a [22] http://cs231n.github.io/classification/
-知识共享许可协议
本教程 由 PaddlePaddle 创作，采用 知识共享 署名-非商业性使用-相同方式共享 4.0 国际 许可协议进行许可。 +This tutorial is contributed by PaddlePaddle, and licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. diff --git a/image_classification/README.ipynb b/image_classification/README.ipynb new file mode 100644 index 0000000000000000000000000000000000000000..e542f32c1fedba5ccd12cc04ad053aa5df4a0dec --- /dev/null +++ b/image_classification/README.ipynb @@ -0,0 +1,877 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# 图像分类\n", + "\n", + "本教程源代码目录在[book/image_classification](https://github.com/PaddlePaddle/book/tree/develop/image_classification)， 初次使用请参考PaddlePaddle[安装教程](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/getstarted/build_and_install/docker_install_cn.rst)。\n", + "\n", + "## 背景介绍\n", + "\n", + "图像相比文字能够提供更加生动、容易理解及更具艺术感的信息，是人们转递与交换信息的重要来源。在本教程中，我们专注于图像识别领域的一个重要问题，即图像分类。\n", + "\n", + "图像分类是根据图像的语义信息将不同类别图像区分开来，是计算机视觉中重要的基本问题，也是图像检测、图像分割、物体跟踪、行为分析等其他高层视觉任务的基础。图像分类在很多领域有广泛应用，包括安防领域的人脸识别和智能视频分析等，交通领域的交通场景识别，互联网领域基于内容的图像检索和相册自动归类，医学领域的图像识别等。\n", + "\n", + "\n", + "一般来说，图像分类通过手工特征或特征学习方法对整个图像进行全部描述，然后使用分类器判别物体类别，因此如何提取图像的特征至关重要。在深度学习算法之前使用较多的是基于词袋(Bag of Words)模型的物体分类方法。词袋方法从自然语言处理中引入，即一句话可以用一个装了词的袋子表示其特征，袋子中的词为句子中的单词、短语或字。对于图像而言，词袋方法需要构建字典。最简单的词袋模型框架可以设计为**底层特征抽取**、**特征编码**、**分类器设计**三个过程。\n", + "\n", + "而基于深度学习的图像分类方法，可以通过有监督或无监督的方式**学习**层次化的特征描述，从而取代了手工设计或选择图像特征的工作。深度学习模型中的卷积神经网络(Convolution Neural Network, CNN)近年来在图像领域取得了惊人的成绩，CNN直接利用图像像素信息作为输入，最大程度上保留了输入图像的所有信息，通过卷积操作进行特征的提取和高层抽象，模型输出直接是图像识别的结果。这种基于\"输入-输出\"直接端到端的学习方法取得了非常好的效果，得到了广泛的应用。\n", + "\n", + "本教程主要介绍图像分类的深度学习模型，以及如何使用PaddlePaddle训练CNN模型。\n", + "\n", + "## 效果展示\n", + "\n", + "图像分类包括通用图像分类、细粒度图像分类等。图1展示了通用图像分类效果，即模型可以正确识别图像上的主要物体。\n", + "\n", + "\u003cp align=\"center\"\u003e\n", + "\u003cimg src=\"image/dog_cat.png \" width=\"350\" \u003e\u003cbr/\u003e\n", + "图1. 通用图像分类展示\n", + "\u003c/p\u003e\n", + "\n", + "\n", + "图2展示了细粒度图像分类-花卉识别的效果，要求模型可以正确识别花的类别。\n", + "\n", + "\n", + "\u003cp align=\"center\"\u003e\n", + "\u003cimg src=\"image/flowers.png\" width=\"400\" \u003e\u003cbr/\u003e\n", + "图2. 细粒度图像分类展示\n", + "\u003c/p\u003e\n", + "\n", + "\n", + "一个好的模型既要对不同类别识别正确，同时也应该能够对不同视角、光照、背景、变形或部分遮挡的图像正确识别(这里我们统一称作图像扰动)。图3展示了一些图像的扰动，较好的模型会像聪明的人类一样能够正确识别。\n", + "\n", + "\u003cp align=\"center\"\u003e\n", + "\u003cimg src=\"image/variations.png\" width=\"550\" \u003e\u003cbr/\u003e\n", + "图3. 扰动图片展示[22]\n", + "\u003c/p\u003e\n", + "\n", + "## 模型概览\n", + "\n", + "图像识别领域大量的研究成果都是建立在[PASCAL VOC](http://host.robots.ox.ac.uk/pascal/VOC/)、[ImageNet](http://image-net.org/)等公开的数据集上，很多图像识别算法通常在这些数据集上进行测试和比较。PASCAL VOC是2005年发起的一个视觉挑战赛，ImageNet是2010年发起的大规模视觉识别竞赛(ILSVRC)的数据集，在本章中我们基于这些竞赛的一些论文介绍图像分类模型。\n", + "\n", + "在2012年之前的传统图像分类方法可以用背景描述中提到的三步完成，但通常完整建立图像识别模型一般包括底层特征学习、特征编码、空间约束、分类器设计、模型融合等几个阶段。\n", + " 1). **底层特征提取**: 通常从图像中按照固定步长、尺度提取大量局部特征描述。常用的局部特征包括SIFT(Scale-Invariant Feature Transform, 尺度不变特征转换) \\[[1](#参考文献)\\]、HOG(Histogram of Oriented Gradient, 方向梯度直方图) \\[[2](#参考文献)\\]、LBP(Local Bianray Pattern, 局部二值模式) \\[[3](#参考文献)\\] 等，一般也采用多种特征描述子，防止丢失过多的有用信息。\n", + " 2). **特征编码**: 底层特征中包含了大量冗余与噪声，为了提高特征表达的鲁棒性，需要使用一种特征变换算法对底层特征进行编码，称作特征编码。常用的特征编码包括向量量化编码 \\[[4](#参考文献)\\]、稀疏编码 \\[[5](#参考文献)\\]、局部线性约束编码 \\[[6](#参考文献)\\]、Fisher向量编码 \\[[7](#参考文献)\\] 等。\n", + " 3). **空间特征约束**: 特征编码之后一般会经过空间特征约束，也称作**特征汇聚**。特征汇聚是指在一个空间范围内，对每一维特征取最大值或者平均值，可以获得一定特征不变形的特征表达。金字塔特征匹配是一种常用的特征聚会方法，这种方法提出将图像均匀分块，在分块内做特征汇聚。\n", + " 4). **通过分类器分类**: 经过前面步骤之后一张图像可以用一个固定维度的向量进行描述，接下来就是经过分类器对图像进行分类。通常使用的分类器包括SVM(Support Vector Machine, 支持向量机)、随机森林等。而使用核方法的SVM是最为广泛的分类器，在传统图像分类任务上性能很好。\n", + "\n", + "这种方法在PASCAL VOC竞赛中的图像分类算法中被广泛使用 \\[[18](#参考文献)\\]。[NEC实验室](http://www.nec-labs.com/)在ILSVRC2010中采用SIFT和LBP特征，两个非线性编码器以及SVM分类器获得图像分类的冠军 \\[[8](#参考文献)\\]。\n", + "\n", + "Alex Krizhevsky在2012年ILSVRC提出的CNN模型 \\[[9](#参考文献)\\] 取得了历史性的突破，效果大幅度超越传统方法，获得了ILSVRC2012冠军，该模型被称作AlexNet。这也是首次将深度学习用于大规模图像分类中。从AlexNet之后，涌现了一系列CNN模型，不断地在ImageNet上刷新成绩，如图4展示。随着模型变得越来越深以及精妙的结构设计，Top-5的错误率也越来越低，降到了3.5%附近。而在同样的ImageNet数据集上，人眼的辨识错误率大概在5.1%，也就是目前的深度学习模型的识别能力已经超过了人眼。\n", + "\n", + "\u003cp align=\"center\"\u003e\n", + "\u003cimg src=\"image/ilsvrc.png\" width=\"500\" \u003e\u003cbr/\u003e\n", + "图4. ILSVRC图像分类Top-5错误率\n", + "\u003c/p\u003e\n", + "\n", + "### CNN\n", + "\n", + "传统CNN包含卷积层、全连接层等组件，并采用softmax多类别分类器和多类交叉熵损失函数，一个典型的卷积神经网络如图5所示，我们先介绍用来构造CNN的常见组件。\n", + "\n", + "\u003cp align=\"center\"\u003e\n", + "\u003cimg src=\"image/lenet.png\"\u003e\u003cbr/\u003e\n", + "图5. CNN网络示例[20]\n", + "\u003c/p\u003e\n", + "\n", + "- 卷积层(convolution layer): 执行卷积操作提取底层到高层的特征，发掘出图片局部关联性质和空间不变性质。\n", + "- 池化层(pooling layer): 执行降采样操作。通过取卷积输出特征图中局部区块的最大值(max-pooling)或者均值(avg-pooling)。降采样也是图像处理中常见的一种操作，可以过滤掉一些不重要的高频信息。\n", + "- 全连接层(fully-connected layer，或者fc layer): 输入层到隐藏层的神经元是全部连接的。\n", + "- 非线性变化: 卷积层、全连接层后面一般都会接非线性变化层，例如Sigmoid、Tanh、ReLu等来增强网络的表达能力，在CNN里最常使用的为ReLu激活函数。\n", + "- Dropout \\[[10](#参考文献)\\] : 在模型训练阶段随机让一些隐层节点权重不工作，提高网络的泛化能力，一定程度上防止过拟合。\n", + "\n", + "另外，在训练过程中由于每层参数不断更新，会导致下一次输入分布发生变化，这样导致训练过程需要精心设计超参数。如2015年Sergey Ioffe和Christian Szegedy提出了Batch Normalization (BN)算法 \\[[14](#参考文献)\\] 中，每个batch对网络中的每一层特征都做归一化，使得每层分布相对稳定。BN算法不仅起到一定的正则作用，而且弱化了一些超参数的设计。经过实验证明，BN算法加速了模型收敛过程，在后来较深的模型中被广泛使用。\n", + "\n", + "接下来我们主要介绍VGG，GoogleNet和ResNet网络结构。\n", + "\n", + "### VGG\n", + "\n", + "牛津大学VGG(Visual Geometry Group)组在2014年ILSVRC提出的模型被称作VGG模型 \\[[11](#参考文献)\\] 。该模型相比以往模型进一步加宽和加深了网络结构，它的核心是五组卷积操作，每两组之间做Max-Pooling空间降维。同一组内采用多次连续的3X3卷积，卷积核的数目由较浅组的64增多到最深组的512，同一组内的卷积核数目是一样的。卷积之后接两层全连接层，之后是分类层。由于每组内卷积层的不同，有11、13、16、19层这几种模型，下图展示一个16层的网络结构。VGG模型结构相对简洁，提出之后也有很多文章基于此模型进行研究，如在ImageNet上首次公开超过人眼识别的模型\\[[19](#参考文献)\\]就是借鉴VGG模型的结构。\n", + "\n", + "\u003cp align=\"center\"\u003e\n", + "\u003cimg src=\"image/vgg16.png\" width=\"750\" \u003e\u003cbr/\u003e\n", + "图6. 基于ImageNet的VGG16模型\n", + "\u003c/p\u003e\n", + "\n", + "### GoogleNet\n", + "\n", + "GoogleNet \\[[12](#参考文献)\\] 在2014年ILSVRC的获得了冠军，在介绍该模型之前我们先来了解NIN(Network in Network)模型 \\[[13](#参考文献)\\] 和Inception模块，因为GoogleNet模型由多组Inception模块组成，模型设计借鉴了NIN的一些思想。\n", + "\n", + "NIN模型主要有两个特点：1) 引入了多层感知卷积网络(Multi-Layer Perceptron Convolution, MLPconv)代替一层线性卷积网络。MLPconv是一个微小的多层卷积网络，即在线性卷积后面增加若干层1x1的卷积，这样可以提取出高度非线性特征。2) 传统的CNN最后几层一般都是全连接层，参数较多。而NIN模型设计最后一层卷积层包含类别维度大小的特征图，然后采用全局均值池化(Avg-Pooling)替代全连接层，得到类别维度大小的向量，再进行分类。这种替代全连接层的方式有利于减少参数。\n", + "\n", + "Inception模块如下图7所示，图(a)是最简单的设计，输出是3个卷积层和一个池化层的特征拼接。这种设计的缺点是池化层不会改变特征通道数，拼接后会导致特征的通道数较大，经过几层这样的模块堆积后，通道数会越来越大，导致参数和计算量也随之增大。为了改善这个缺点，图(b)引入3个1x1卷积层进行降维，所谓的降维就是减少通道数，同时如NIN模型中提到的1x1卷积也可以修正线性特征。\n", + "\n", + "\u003cp align=\"center\"\u003e\n", + "\u003cimg src=\"image/inception.png\" width=\"800\" \u003e\u003cbr/\u003e\n", + "图7. Inception模块\n", + "\u003c/p\u003e\n", + "\n", + "GoogleNet由多组Inception模块堆积而成。另外，在网络最后也没有采用传统的多层全连接层，而是像NIN网络一样采用了均值池化层；但与NIN不同的是，池化层后面接了一层到类别数映射的全连接层。除了这两个特点之外，由于网络中间层特征也很有判别性，GoogleNet在中间层添加了两个辅助分类器，在后向传播中增强梯度并且增强正则化，而整个网络的损失函数是这个三个分类器的损失加权求和。\n", + "\n", + "GoogleNet整体网络结构如图8所示，总共22层网络：开始由3层普通的卷积组成；接下来由三组子网络组成，第一组子网络包含2个Inception模块，第二组包含5个Inception模块，第三组包含2个Inception模块；然后接均值池化层、全连接层。\n", + "\n", + "\u003cp align=\"center\"\u003e\n", + "\u003cimg src=\"image/googlenet.jpeg\" \u003e\u003cbr/\u003e\n", + "图8. GoogleNet[12]\n", + "\u003c/p\u003e\n", + "\n", + "\n", + "上面介绍的是GoogleNet第一版模型(称作GoogleNet-v1)。GoogleNet-v2 \\[[14](#参考文献)\\] 引入BN层；GoogleNet-v3 \\[[16](#参考文献)\\] 对一些卷积层做了分解，进一步提高网络非线性能力和加深网络；GoogleNet-v4 \\[[17](#参考文献)\\] 引入下面要讲的ResNet设计思路。从v1到v4每一版的改进都会带来准确度的提升，介于篇幅，这里不再详细介绍v2到v4的结构。\n", + "\n", + "\n", + "### ResNet\n", + "\n", + "ResNet(Residual Network) \\[[15](#参考文献)\\] 是2015年ImageNet图像分类、图像物体定位和图像物体检测比赛的冠军。针对训练卷积神经网络时加深网络导致准确度下降的问题，ResNet提出了采用残差学习。在已有设计思路(BN, 小卷积核，全卷积网络)的基础上，引入了残差模块。每个残差模块包含两条路径，其中一条路径是输入特征的直连通路，另一条路径对该特征做两到三次卷积操作得到该特征的残差，最后再将两条路径上的特征相加。\n", + "\n", + "残差模块如图9所示，左边是基本模块连接方式，由两个输出通道数相同的3x3卷积组成。右边是瓶颈模块(Bottleneck)连接方式，之所以称为瓶颈，是因为上面的1x1卷积用来降维(图示例即256-\u003e64)，下面的1x1卷积用来升维(图示例即64-\u003e256)，这样中间3x3卷积的输入和输出通道数都较小(图示例即64-\u003e64)。\n", + "\n", + "\u003cp align=\"center\"\u003e\n", + "\u003cimg src=\"image/resnet_block.jpg\" width=\"400\"\u003e\u003cbr/\u003e\n", + "图9. 残差模块\n", + "\u003c/p\u003e\n", + "\n", + "图10展示了50、101、152层网络连接示意图，使用的是瓶颈模块。这三个模型的区别在于每组中残差模块的重复次数不同(见图右上角)。ResNet训练收敛较快，成功的训练了上百乃至近千层的卷积神经网络。\n", + "\n", + "\u003cp align=\"center\"\u003e\n", + "\u003cimg src=\"image/resnet.png\"\u003e\u003cbr/\u003e\n", + "图10. 基于ImageNet的ResNet模型\n", + "\u003c/p\u003e\n", + "\n", + "\n", + "## 数据准备\n", + "\n", + "通用图像分类公开的标准数据集常用的有[CIFAR](\u003chttps://www.cs.toronto.edu/~kriz/cifar.html)、[ImageNet](http://image-net.org/)、[COCO](http://mscoco.org/)等，常用的细粒度图像分类数据集包括[CUB-200-2011](http://www.vision.caltech.edu/visipedia/CUB-200-2011.html)、[Stanford Dog](http://vision.stanford.edu/aditya86/ImageNetDogs/)、[Oxford-flowers](http://www.robots.ox.ac.uk/~vgg/data/flowers/)等。其中ImageNet数据集规模相对较大，如[模型概览](#模型概览)一章所讲，大量研究成果基于ImageNet。ImageNet数据从2010年来稍有变化，常用的是ImageNet-2012数据集，该数据集包含1000个类别：训练集包含1,281,167张图片，每个类别数据732至1300张不等，验证集包含50,000张图片，平均每个类别50张图片。\n", + "\n", + "由于ImageNet数据集较大，下载和训练较慢，为了方便大家学习，我们使用[CIFAR10](\u003chttps://www.cs.toronto.edu/~kriz/cifar.html\u003e)数据集。CIFAR10数据集包含60,000张32x32的彩色图片，10个类别，每个类包含6,000张。其中50,000张图片作为训练集，10000张作为测试集。图11从每个类别中随机抽取了10张图片，展示了所有的类别。\n", + "\n", + "\u003cp align=\"center\"\u003e\n", + "\u003cimg src=\"image/cifar.png\" width=\"350\"\u003e\u003cbr/\u003e\n", + "图11. CIFAR10数据集[21]\n", + "\u003c/p\u003e\n", + "\n", + "Paddle API提供了自动加载cifar数据集模块 `paddle.dataset.cifar`。\n", + "\n", + "通过输入`python train.py`，就可以开始训练模型了，以下小节将详细介绍`train.py`的相关内容。\n", + "\n", + "### 模型结构\n", + "\n", + "#### Paddle 初始化\n", + "\n", + "通过 `paddle.init`，初始化Paddle是否使用GPU，trainer的数目等等。\n", + "\n" + ] + }, + { + "cell_type": "code", + "metadata": { + "editable": true + }, + "source": [ + "import sys\n", + "import paddle.v2 as paddle\n", + "from vgg import vgg_bn_drop\n", + "from resnet import resnet_cifar10\n", + "\n", + "# PaddlePaddle init\n", + "paddle.init(use_gpu=False, trainer_count=1)\n" + ], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n" + ] + } + ], + "execution_count": 1 + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "本教程中我们提供了VGG和ResNet两个模型的配置。\n", + "\n", + "#### VGG\n", + "\n", + "首先介绍VGG模型结构，由于CIFAR10图片大小和数量相比ImageNet数据小很多，因此这里的模型针对CIFAR10数据做了一定的适配。卷积部分引入了BN和Dropout操作。\n", + "\n", + "1. 定义数据输入及其维度\n", + "\n", + " 网络输入定义为 `data_layer` (数据层)，在图像分类中即为图像像素信息。CIFRAR10是RGB 3通道32x32大小的彩色图，因此输入数据大小为3072(3x32x32)，类别大小为10，即10分类。\n", + "\n" + ] + }, + { + "cell_type": "code", + "metadata": { + "editable": true + }, + "source": [ + " datadim = 3 * 32 * 32\n", + " classdim = 10\n", + "\n", + " image = paddle.layer.data(\n", + " name=\"image\", type=paddle.data_type.dense_vector(datadim))\n" + ], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n" + ] + } + ], + "execution_count": 1 + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "2. 定义VGG网络核心模块\n", + "\n" + ] + }, + { + "cell_type": "code", + "metadata": { + "editable": true + }, + "source": [ + " net = vgg_bn_drop(image)\n" + ], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n" + ] + } + ], + "execution_count": 1 + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + " VGG核心模块的输入是数据层，`vgg_bn_drop` 定义了16层VGG结构，每层卷积后面引入BN层和Dropout层，详细的定义如下：\n", + "\n" + ] + }, + { + "cell_type": "code", + "metadata": { + "editable": true + }, + "source": [ + " def vgg_bn_drop(input):\n", + " def conv_block(ipt, num_filter, groups, dropouts, num_channels=None):\n", + " return paddle.networks.img_conv_group(\n", + " input=ipt,\n", + " num_channels=num_channels,\n", + " pool_size=2,\n", + " pool_stride=2,\n", + " conv_num_filter=[num_filter] * groups,\n", + " conv_filter_size=3,\n", + " conv_act=paddle.activation.Relu(),\n", + " conv_with_batchnorm=True,\n", + " conv_batchnorm_drop_rate=dropouts,\n", + " pool_type=paddle.pooling.Max())\n", + "\n", + " conv1 = conv_block(input, 64, 2, [0.3, 0], 3)\n", + " conv2 = conv_block(conv1, 128, 2, [0.4, 0])\n", + " conv3 = conv_block(conv2, 256, 3, [0.4, 0.4, 0])\n", + " conv4 = conv_block(conv3, 512, 3, [0.4, 0.4, 0])\n", + " conv5 = conv_block(conv4, 512, 3, [0.4, 0.4, 0])\n", + "\n", + " drop = paddle.layer.dropout(input=conv5, dropout_rate=0.5)\n", + " fc1 = paddle.layer.fc(input=drop, size=512, act=paddle.activation.Linear())\n", + " bn = paddle.layer.batch_norm(\n", + " input=fc1,\n", + " act=paddle.activation.Relu(),\n", + " layer_attr=paddle.attr.Extra(drop_rate=0.5))\n", + " fc2 = paddle.layer.fc(input=bn, size=512, act=paddle.activation.Linear())\n", + " return fc2\n" + ], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n" + ] + } + ], + "execution_count": 1 + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + " 2.1. 首先定义了一组卷积网络，即conv_block。卷积核大小为3x3，池化窗口大小为2x2，窗口滑动大小为2，groups决定每组VGG模块是几次连续的卷积操作，dropouts指定Dropout操作的概率。所使用的`img_conv_group`是在`paddle.networks`中预定义的模块，由若干组 `Conv-\u003eBN-\u003eReLu-\u003eDropout` 和 一组 `Pooling` 组成，\n", + "\n", + " 2.2. 五组卷积操作，即 5个conv_block。 第一、二组采用两次连续的卷积操作。第三、四、五组采用三次连续的卷积操作。每组最后一个卷积后面Dropout概率为0，即不使用Dropout操作。\n", + "\n", + " 2.3. 最后接两层512维的全连接。\n", + "\n", + "3. 定义分类器\n", + "\n", + " 通过上面VGG网络提取高层特征，然后经过全连接层映射到类别维度大小的向量，再通过Softmax归一化得到每个类别的概率，也可称作分类器。\n", + "\n" + ] + }, + { + "cell_type": "code", + "metadata": { + "editable": true + }, + "source": [ + " out = paddle.layer.fc(input=net,\n", + " size=classdim,\n", + " act=paddle.activation.Softmax())\n" + ], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n" + ] + } + ], + "execution_count": 1 + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "4. 定义损失函数和网络输出\n", + "\n", + " 在有监督训练中需要输入图像对应的类别信息，同样通过`paddle.layer.data`来定义。训练中采用多类交叉熵作为损失函数，并作为网络的输出，预测阶段定义网络的输出为分类器得到的概率信息。\n", + "\n" + ] + }, + { + "cell_type": "code", + "metadata": { + "editable": true + }, + "source": [ + " lbl = paddle.layer.data(\n", + " name=\"label\", type=paddle.data_type.integer_value(classdim))\n", + " cost = paddle.layer.classification_cost(input=out, label=lbl)\n" + ], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n" + ] + } + ], + "execution_count": 1 + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "### ResNet\n", + "\n", + "ResNet模型的第1、3、4步和VGG模型相同，这里不再介绍。主要介绍第2步即CIFAR10数据集上ResNet核心模块。\n", + "\n" + ] + }, + { + "cell_type": "code", + "metadata": { + "editable": true + }, + "source": [ + "net = resnet_cifar10(image, depth=56)\n" + ], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n" + ] + } + ], + "execution_count": 1 + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "先介绍`resnet_cifar10`中的一些基本函数，再介绍网络连接过程。\n", + "\n", + " - `conv_bn_layer` : 带BN的卷积层。\n", + " - `shortcut` : 残差模块的\"直连\"路径，\"直连\"实际分两种形式：残差模块输入和输出特征通道数不等时，采用1x1卷积的升维操作；残差模块输入和输出通道相等时，采用直连操作。\n", + " - `basicblock` : 一个基础残差模块，即图9左边所示，由两组3x3卷积组成的路径和一条\"直连\"路径组成。\n", + " - `bottleneck` : 一个瓶颈残差模块，即图9右边所示，由上下1x1卷积和中间3x3卷积组成的路径和一条\"直连\"路径组成。\n", + " - `layer_warp` : 一组残差模块，由若干个残差模块堆积而成。每组中第一个残差模块滑动窗口大小与其他可以不同，以用来减少特征图在垂直和水平方向的大小。\n", + "\n" + ] + }, + { + "cell_type": "code", + "metadata": { + "editable": true + }, + "source": [ + "def conv_bn_layer(input,\n", + " ch_out,\n", + " filter_size,\n", + " stride,\n", + " padding,\n", + " active_type=paddle.activation.Relu(),\n", + " ch_in=None):\n", + " tmp = paddle.layer.img_conv(\n", + " input=input,\n", + " filter_size=filter_size,\n", + " num_channels=ch_in,\n", + " num_filters=ch_out,\n", + " stride=stride,\n", + " padding=padding,\n", + " act=paddle.activation.Linear(),\n", + " bias_attr=False)\n", + " return paddle.layer.batch_norm(input=tmp, act=active_type)\n", + "\n", + "def shortcut(ipt, n_in, n_out, stride):\n", + " if n_in != n_out:\n", + " return conv_bn_layer(ipt, n_out, 1, stride, 0,\n", + " paddle.activation.Linear())\n", + " else:\n", + " return ipt\n", + "\n", + "def basicblock(ipt, ch_out, stride):\n", + " ch_in = ch_out * 2\n", + " tmp = conv_bn_layer(ipt, ch_out, 3, stride, 1)\n", + " tmp = conv_bn_layer(tmp, ch_out, 3, 1, 1, paddle.activation.Linear())\n", + " short = shortcut(ipt, ch_in, ch_out, stride)\n", + " return paddle.layer.addto(input=[tmp, short], act=paddle.activation.Relu())\n", + "\n", + "def layer_warp(block_func, ipt, features, count, stride):\n", + " tmp = block_func(ipt, features, stride)\n", + " for i in range(1, count):\n", + " tmp = block_func(tmp, features, 1)\n", + " return tmp\n" + ], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n" + ] + } + ], + "execution_count": 1 + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "`resnet_cifar10` 的连接结构主要有以下几个过程。\n", + "\n", + "1. 底层输入连接一层 `conv_bn_layer`，即带BN的卷积层。\n", + "2. 然后连接3组残差模块即下面配置3组 `layer_warp` ，每组采用图 10 左边残差模块组成。\n", + "3. 最后对网络做均值池化并返回该层。\n", + "\n", + "注意：除过第一层卷积层和最后一层全连接层之外，要求三组 `layer_warp` 总的含参层数能够被6整除，即 `resnet_cifar10` 的 depth 要满足 $(depth - 2) % 6 == 0$ 。\n", + "\n" + ] + }, + { + "cell_type": "code", + "metadata": { + "editable": true + }, + "source": [ + "def resnet_cifar10(ipt, depth=32):\n", + " # depth should be one of 20, 32, 44, 56, 110, 1202\n", + " assert (depth - 2) % 6 == 0\n", + " n = (depth - 2) / 6\n", + " nStages = {16, 64, 128}\n", + " conv1 = conv_bn_layer(\n", + " ipt, ch_in=3, ch_out=16, filter_size=3, stride=1, padding=1)\n", + " res1 = layer_warp(basicblock, conv1, 16, n, 1)\n", + " res2 = layer_warp(basicblock, res1, 32, n, 2)\n", + " res3 = layer_warp(basicblock, res2, 64, n, 2)\n", + " pool = paddle.layer.img_pool(\n", + " input=res3, pool_size=8, stride=1, pool_type=paddle.pooling.Avg())\n", + " return pool\n" + ], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n" + ] + } + ], + "execution_count": 1 + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "## 训练模型\n", + "\n", + "### 定义参数\n", + "\n", + "首先依据模型配置的`cost`定义模型参数。\n", + "\n" + ] + }, + { + "cell_type": "code", + "metadata": { + "editable": true + }, + "source": [ + "# Create parameters\n", + "parameters = paddle.parameters.create(cost)\n" + ], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n" + ] + } + ], + "execution_count": 1 + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "可以打印参数名字，如果在网络配置中没有指定名字，则默认生成。\n", + "\n" + ] + }, + { + "cell_type": "code", + "metadata": { + "editable": true + }, + "source": [ + "print parameters.keys()\n" + ], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n" + ] + } + ], + "execution_count": 1 + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "### 构造训练(Trainer)\n", + "\n", + "根据网络拓扑结构和模型参数来构造出trainer用来训练，在构造时还需指定优化方法，这里使用最基本的Momentum方法，同时设定了学习率、正则等。\n", + "\n" + ] + }, + { + "cell_type": "code", + "metadata": { + "editable": true + }, + "source": [ + "# Create optimizer\n", + "momentum_optimizer = paddle.optimizer.Momentum(\n", + " momentum=0.9,\n", + " regularization=paddle.optimizer.L2Regularization(rate=0.0002 * 128),\n", + " learning_rate=0.1 / 128.0,\n", + " learning_rate_decay_a=0.1,\n", + " learning_rate_decay_b=50000 * 100,\n", + " learning_rate_schedule='discexp',\n", + " batch_size=128)\n", + "\n", + "# Create trainer\n", + "trainer = paddle.trainer.SGD(cost=cost,\n", + " parameters=parameters,\n", + " update_equation=momentum_optimizer)\n" + ], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n" + ] + } + ], + "execution_count": 1 + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "通过 `learning_rate_decay_a` (简写$a$） 、`learning_rate_decay_b` (简写$b$) 和 `learning_rate_schedule` 指定学习率调整策略，这里采用离散指数的方式调节学习率，计算公式如下， $n$ 代表已经处理过的累计总样本数，$lr_{0}$ 即为 `settings` 里设置的 `learning_rate`。\n", + "\n", + "$$ lr = lr_{0} * a^ {\\lfloor \\frac{n}{ b}\\rfloor} $$\n", + "\n", + "\n", + "### 训练\n", + "\n", + "cifar.train10()每次产生一条样本，在完成shuffle和batch之后，作为训练的输入。\n", + "\n" + ] + }, + { + "cell_type": "code", + "metadata": { + "editable": true + }, + "source": [ + "reader=paddle.batch(\n", + " paddle.reader.shuffle(\n", + " paddle.dataset.cifar.train10(), buf_size=50000),\n", + " batch_size=128)\n" + ], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n" + ] + } + ], + "execution_count": 1 + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "通过`feeding`来指定每一个数据和`paddle.layer.data`的对应关系。例如: `cifar.train10()`产生数据的第0列对应image层的特征。\n", + "\n" + ] + }, + { + "cell_type": "code", + "metadata": { + "editable": true + }, + "source": [ + "feeding={'image': 0,\n", + " 'label': 1}\n" + ], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n" + ] + } + ], + "execution_count": 1 + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "可以使用`event_handler`回调函数来观察训练过程，或进行测试等, 该回调函数是`trainer.train`函数里设定。\n", + "\n" + ] + }, + { + "cell_type": "code", + "metadata": { + "editable": true + }, + "source": [ + "# End batch and end pass event handler\n", + "def event_handler(event):\n", + " if isinstance(event, paddle.event.EndIteration):\n", + " if event.batch_id % 100 == 0:\n", + " print \"\\nPass %d, Batch %d, Cost %f, %s\" % (\n", + " event.pass_id, event.batch_id, event.cost, event.metrics)\n", + " else:\n", + " sys.stdout.write('.')\n", + " sys.stdout.flush()\n", + " if isinstance(event, paddle.event.EndPass):\n", + " result = trainer.test(\n", + " reader=paddle.batch(\n", + " paddle.dataset.cifar.test10(), batch_size=128),\n", + " feeding=feeding)\n", + " print \"\\nTest with Pass %d, %s\" % (event.pass_id, result.metrics)\n" + ], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n" + ] + } + ], + "execution_count": 1 + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "通过`trainer.train`函数训练:\n", + "\n" + ] + }, + { + "cell_type": "code", + "metadata": { + "editable": true + }, + "source": [ + "trainer.train(\n", + " reader=reader,\n", + " num_passes=200,\n", + " event_handler=event_handler,\n", + " feeding=feeding)\n" + ], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n" + ] + } + ], + "execution_count": 1 + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "一轮训练log示例如下所示，经过1个pass， 训练集上平均error为0.6875 ，测试集上平均error为0.8852 。\n", + "\n", + "```text\n", + "Pass 0, Batch 0, Cost 2.473182, {'classification_error_evaluator': 0.9140625}\n", + "...................................................................................................\n", + "Pass 0, Batch 100, Cost 1.913076, {'classification_error_evaluator': 0.78125}\n", + "...................................................................................................\n", + "Pass 0, Batch 200, Cost 1.783041, {'classification_error_evaluator': 0.7421875}\n", + "...................................................................................................\n", + "Pass 0, Batch 300, Cost 1.668833, {'classification_error_evaluator': 0.6875}\n", + "..........................................................................................\n", + "Test with Pass 0, {'classification_error_evaluator': 0.885200023651123}\n", + "```\n", + "\n", + "图12是训练的分类错误率曲线图，运行到第200个pass后基本收敛，最终得到测试集上分类错误率为8.54%。\n", + "\n", + "\u003cp align=\"center\"\u003e\n", + "\u003cimg src=\"image/plot.png\" width=\"400\" \u003e\u003cbr/\u003e\n", + "图12. CIFAR10数据集上VGG模型的分类错误率\n", + "\u003c/p\u003e\n", + "\n", + "\n", + "## 总结\n", + "\n", + "传统图像分类方法由多个阶段构成，框架较为复杂，而端到端的CNN模型结构可一步到位，而且大幅度提升了分类准确率。本文我们首先介绍VGG、GoogleNet、ResNet三个经典的模型；然后基于CIFAR10数据集，介绍如何使用PaddlePaddle配置和训练CNN模型，尤其是VGG和ResNet模型；最后介绍如何使用PaddlePaddle的API接口对图片进行预测和特征提取。对于其他数据集比如ImageNet，配置和训练流程是同样的，大家可以自行进行实验。\n", + "\n", + "\n", + "## 参考文献\n", + "\n", + "[1] D. G. Lowe, [Distinctive image features from scale-invariant keypoints](http://www.cs.ubc.ca/~lowe/papers/ijcv04.pdf). IJCV, 60(2):91-110, 2004.\n", + "\n", + "[2] N. Dalal, B. Triggs, [Histograms of Oriented Gradients for Human Detection](http://vision.stanford.edu/teaching/cs231b_spring1213/papers/CVPR05_DalalTriggs.pdf), Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2005.\n", + "\n", + "[3] Ahonen, T., Hadid, A., and Pietikinen, M. (2006). [Face description with local binary patterns: Application to face recognition](http://ieeexplore.ieee.org/document/1717463/). PAMI, 28.\n", + "\n", + "[4] J. Sivic, A. Zisserman, [Video Google: A Text Retrieval Approach to Object Matching in Videos](http://www.robots.ox.ac.uk/~vgg/publications/papers/sivic03.pdf), Proc. Ninth Int'l Conf. Computer Vision, pp. 1470-1478, 2003.\n", + "\n", + "[5] B. Olshausen, D. Field, [Sparse Coding with an Overcomplete Basis Set: A Strategy Employed by V1?](http://redwood.psych.cornell.edu/papers/olshausen_field_1997.pdf), Vision Research, vol. 37, pp. 3311-3325, 1997.\n", + "\n", + "[6] Wang, J., Yang, J., Yu, K., Lv, F., Huang, T., and Gong, Y. (2010). [Locality-constrained Linear Coding for image classification](http://ieeexplore.ieee.org/abstract/document/5540018/). In CVPR.\n", + "\n", + "[7] Perronnin, F., Sánchez, J., \u0026 Mensink, T. (2010). [Improving the fisher kernel for large-scale image classification](http://dl.acm.org/citation.cfm?id=1888101). In ECCV (4).\n", + "\n", + "[8] Lin, Y., Lv, F., Cao, L., Zhu, S., Yang, M., Cour, T., Yu, K., and Huang, T. (2011). [Large-scale image clas- sification: Fast feature extraction and SVM training](http://ieeexplore.ieee.org/document/5995477/). In CVPR.\n", + "\n", + "[9] Krizhevsky, A., Sutskever, I., and Hinton, G. (2012). [ImageNet classification with deep convolutional neu- ral networks](http://www.cs.toronto.edu/~kriz/imagenet_classification_with_deep_convolutional.pdf). In NIPS.\n", + "\n", + "[10] G.E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R.R. Salakhutdinov. [Improving neural networks by preventing co-adaptation of feature detectors](https://arxiv.org/abs/1207.0580). arXiv preprint arXiv:1207.0580, 2012.\n", + "\n", + "[11] K. Chatfield, K. Simonyan, A. Vedaldi, A. Zisserman. [Return of the Devil in the Details: Delving Deep into Convolutional Nets](https://arxiv.org/abs/1405.3531). BMVC, 2014。\n", + "\n", + "[12] Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A., [Going deeper with convolutions](https://arxiv.org/abs/1409.4842). In: CVPR. (2015)\n", + "\n", + "[13] Lin, M., Chen, Q., and Yan, S. [Network in network](https://arxiv.org/abs/1312.4400). In Proc. ICLR, 2014.\n", + "\n", + "[14] S. Ioffe and C. Szegedy. [Batch normalization: Accelerating deep network training by reducing internal covariate shift](https://arxiv.org/abs/1502.03167). In ICML, 2015.\n", + "\n", + "[15] K. He, X. Zhang, S. Ren, J. Sun. [Deep Residual Learning for Image Recognition](https://arxiv.org/abs/1512.03385). CVPR 2016.\n", + "\n", + "[16] Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z. [Rethinking the incep-tion architecture for computer vision](https://arxiv.org/abs/1512.00567). In: CVPR. (2016).\n", + "\n", + "[17] Szegedy, C., Ioffe, S., Vanhoucke, V. [Inception-v4, inception-resnet and the impact of residual connections on learning](https://arxiv.org/abs/1602.07261). arXiv:1602.07261 (2016).\n", + "\n", + "[18] Everingham, M., Eslami, S. M. A., Van Gool, L., Williams, C. K. I., Winn, J. and Zisserman, A. [The Pascal Visual Object Classes Challenge: A Retrospective]((http://link.springer.com/article/10.1007/s11263-014-0733-5)). International Journal of Computer Vision, 111(1), 98-136, 2015.\n", + "\n", + "[19] He, K., Zhang, X., Ren, S., and Sun, J. [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://arxiv.org/abs/1502.01852). ArXiv e-prints, February 2015.\n", + "\n", + "[20] http://deeplearning.net/tutorial/lenet.html\n", + "\n", + "[21] https://www.cs.toronto.edu/~kriz/cifar.html\n", + "\n", + "[22] http://cs231n.github.io/classification/\n", + "\n", + "\u003cbr/\u003e\n", + "\u003ca rel=\"license\" href=\"http://creativecommons.org/licenses/by-nc-sa/4.0/\"\u003e\u003cimg alt=\"知识共享许可协议\" style=\"border-width:0\" src=\"https://i.creativecommons.org/l/by-nc-sa/4.0/88x31.png\" /\u003e\u003c/a\u003e\u003cbr /\u003e\u003cspan xmlns:dct=\"http://purl.org/dc/terms/\" href=\"http://purl.org/dc/dcmitype/Text\" property=\"dct:title\" rel=\"dct:type\"\u003e本教程\u003c/span\u003e 由 \u003ca xmlns:cc=\"http://creativecommons.org/ns#\" href=\"http://book.paddlepaddle.org\" property=\"cc:attributionName\" rel=\"cc:attributionURL\"\u003ePaddlePaddle\u003c/a\u003e 创作，采用 \u003ca rel=\"license\" href=\"http://creativecommons.org/licenses/by-nc-sa/4.0/\"\u003e知识共享 署名-非商业性使用-相同方式共享 4.0 国际 许可协议\u003c/a\u003e进行许可。\n" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.6.0" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +} diff --git a/image_classification/README.md b/image_classification/README.md index 829e99a2b9cb9819d87df9bfc53dd81f1a2a6147..c32924b86fa58c04df2ee3766fe87860949ed4da 100644 --- a/image_classification/README.md +++ b/image_classification/README.md @@ -1,7 +1,6 @@ -图像分类 -======= +# 图像分类 -本教程源代码目录在[book/image_classification](https://github.com/PaddlePaddle/book/tree/develop/image_classification)， 初次使用请参考PaddlePaddle[安装教程](http://www.paddlepaddle.org/doc_cn/build_and_install/index.html)。 +本教程源代码目录在[book/image_classification](https://github.com/PaddlePaddle/book/tree/develop/image_classification)， 初次使用请参考PaddlePaddle[安装教程](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/getstarted/build_and_install/docker_install_cn.rst)。 ## 背景介绍 @@ -173,24 +172,24 @@ paddle.init(use_gpu=False, trainer_count=1) 1. 定义数据输入及其维度 - 网络输入定义为 `data_layer` (数据层)，在图像分类中即为图像像素信息。CIFRAR10是RGB 3通道32x32大小的彩色图，因此输入数据大小为3072(3x32x32)，类别大小为10，即10分类。 + 网络输入定义为 `data_layer` (数据层)，在图像分类中即为图像像素信息。CIFRAR10是RGB 3通道32x32大小的彩色图，因此输入数据大小为3072(3x32x32)，类别大小为10，即10分类。 - ```python + ```python datadim = 3 * 32 * 32 classdim = 10 image = paddle.layer.data( name="image", type=paddle.data_type.dense_vector(datadim)) - ``` + ``` 2. 定义VGG网络核心模块 - ```python - net = vgg_bn_drop(image) - ``` - VGG核心模块的输入是数据层，`vgg_bn_drop` 定义了16层VGG结构，每层卷积后面引入BN层和Dropout层，详细的定义如下： + ```python + net = vgg_bn_drop(image) + ``` + VGG核心模块的输入是数据层，`vgg_bn_drop` 定义了16层VGG结构，每层卷积后面引入BN层和Dropout层，详细的定义如下： - ```python + ```python def vgg_bn_drop(input): def conv_block(ipt, num_filter, groups, dropouts, num_channels=None): return paddle.networks.img_conv_group( @@ -219,40 +218,40 @@ paddle.init(use_gpu=False, trainer_count=1) layer_attr=paddle.attr.Extra(drop_rate=0.5)) fc2 = paddle.layer.fc(input=bn, size=512, act=paddle.activation.Linear()) return fc2 - ``` + ``` - 2.1. 首先定义了一组卷积网络，即conv_block。卷积核大小为3x3，池化窗口大小为2x2，窗口滑动大小为2，groups决定每组VGG模块是几次连续的卷积操作，dropouts指定Dropout操作的概率。所使用的`img_conv_group`是在`paddle.networks`中预定义的模块，由若干组 `Conv->BN->ReLu->Dropout` 和 一组 `Pooling` 组成， + 2.1. 首先定义了一组卷积网络，即conv_block。卷积核大小为3x3，池化窗口大小为2x2，窗口滑动大小为2，groups决定每组VGG模块是几次连续的卷积操作，dropouts指定Dropout操作的概率。所使用的`img_conv_group`是在`paddle.networks`中预定义的模块，由若干组 `Conv->BN->ReLu->Dropout` 和 一组 `Pooling` 组成， - 2.2. 五组卷积操作，即 5个conv_block。 第一、二组采用两次连续的卷积操作。第三、四、五组采用三次连续的卷积操作。每组最后一个卷积后面Dropout概率为0，即不使用Dropout操作。 + 2.2. 五组卷积操作，即 5个conv_block。 第一、二组采用两次连续的卷积操作。第三、四、五组采用三次连续的卷积操作。每组最后一个卷积后面Dropout概率为0，即不使用Dropout操作。 - 2.3. 最后接两层512维的全连接。 + 2.3. 最后接两层512维的全连接。 3. 定义分类器 - 通过上面VGG网络提取高层特征，然后经过全连接层映射到类别维度大小的向量，再通过Softmax归一化得到每个类别的概率，也可称作分类器。 + 通过上面VGG网络提取高层特征，然后经过全连接层映射到类别维度大小的向量，再通过Softmax归一化得到每个类别的概率，也可称作分类器。 - ```python + ```python out = paddle.layer.fc(input=net, size=classdim, act=paddle.activation.Softmax()) - ``` + ``` 4. 定义损失函数和网络输出 - 在有监督训练中需要输入图像对应的类别信息，同样通过`paddle.layer.data`来定义。训练中采用多类交叉熵作为损失函数，并作为网络的输出，预测阶段定义网络的输出为分类器得到的概率信息。 + 在有监督训练中需要输入图像对应的类别信息，同样通过`paddle.layer.data`来定义。训练中采用多类交叉熵作为损失函数，并作为网络的输出，预测阶段定义网络的输出为分类器得到的概率信息。 - ```python + ```python lbl = paddle.layer.data( name="label", type=paddle.data_type.integer_value(classdim)) cost = paddle.layer.classification_cost(input=out, label=lbl) - ``` + ``` ### ResNet ResNet模型的第1、3、4步和VGG模型相同，这里不再介绍。主要介绍第2步即CIFAR10数据集上ResNet核心模块。 ```python -net = resnet_cifar10(data, depth=56) +net = resnet_cifar10(image, depth=56) ``` 先介绍`resnet_cifar10`中的一些基本函数，再介绍网络连接过程。 @@ -375,7 +374,7 @@ $$ lr = lr_{0} * a^ {\lfloor \frac{n}{ b}\rfloor} $$ cifar.train10()每次产生一条样本，在完成shuffle和batch之后，作为训练的输入。 ```python -reader=paddle.reader.batch( +reader=paddle.batch( paddle.reader.shuffle( paddle.dataset.cifar.train10(), buf_size=50000), batch_size=128) @@ -402,10 +401,9 @@ def event_handler(event): sys.stdout.flush() if isinstance(event, paddle.event.EndPass): result = trainer.test( - reader=paddle.reader.batch( + reader=paddle.batch( paddle.dataset.cifar.test10(), batch_size=128), - reader_dict={'image': 0, - 'label': 1}) + feeding=feeding) print "\nTest with Pass %d, %s" % (event.pass_id, result.metrics) ``` diff --git a/image_classification/deprecated/README.md b/image_classification/deprecated/README.md index a82a32c8a84cedd5da05e2a66f791819f09f65cb..84125f4ded5e46dfbe15e536ee8de51099469c2b 100644 --- a/image_classification/deprecated/README.md +++ b/image_classification/deprecated/README.md @@ -1,7 +1,7 @@ 图像分类 ======= -本教程源代码目录在[book/image_classification](https://github.com/PaddlePaddle/book/tree/develop/image_classification)， 初次使用请参考PaddlePaddle[安装教程](http://www.paddlepaddle.org/doc_cn/build_and_install/index.html)。 +本教程源代码目录在[book/image_classification](https://github.com/PaddlePaddle/book/tree/develop/image_classification)， 初次使用请参考PaddlePaddle[安装教程](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/getstarted/build_and_install/docker_install_cn.rst)。 ## 背景介绍 @@ -244,77 +244,77 @@ $$ lr = lr_{0} * a^ {\lfloor \frac{n}{ b}\rfloor} $$ 1. 定义数据输入及其维度 - 网络输入定义为 `data_layer` (数据层)，在图像分类中即为图像像素信息。CIFRAR10是RGB 3通道32x32大小的彩色图，因此输入数据大小为3072(3x32x32)，类别大小为10，即10分类。 + 网络输入定义为 `data_layer` (数据层)，在图像分类中即为图像像素信息。CIFRAR10是RGB 3通道32x32大小的彩色图，因此输入数据大小为3072(3x32x32)，类别大小为10，即10分类。 - ```python - datadim = 3 * 32 * 32 - classdim = 10 - data = data_layer(name='image', size=datadim) - ``` + ```python + datadim = 3 * 32 * 32 + classdim = 10 + data = data_layer(name='image', size=datadim) + ``` 2. 定义VGG网络核心模块 - ```python - net = vgg_bn_drop(data) - ``` - VGG核心模块的输入是数据层，`vgg_bn_drop` 定义了16层VGG结构，每层卷积后面引入BN层和Dropout层，详细的定义如下： - - ```python - def vgg_bn_drop(input, num_channels): - def conv_block(ipt, num_filter, groups, dropouts, num_channels_=None): - return img_conv_group( - input=ipt, - num_channels=num_channels_, - pool_size=2, - pool_stride=2, - conv_num_filter=[num_filter] * groups, - conv_filter_size=3, - conv_act=ReluActivation(), - conv_with_batchnorm=True, - conv_batchnorm_drop_rate=dropouts, - pool_type=MaxPooling()) - - conv1 = conv_block(input, 64, 2, [0.3, 0], 3) - conv2 = conv_block(conv1, 128, 2, [0.4, 0]) - conv3 = conv_block(conv2, 256, 3, [0.4, 0.4, 0]) - conv4 = conv_block(conv3, 512, 3, [0.4, 0.4, 0]) - conv5 = conv_block(conv4, 512, 3, [0.4, 0.4, 0]) - - drop = dropout_layer(input=conv5, dropout_rate=0.5) - fc1 = fc_layer(input=drop, size=512, act=LinearActivation()) - bn = batch_norm_layer( - input=fc1, act=ReluActivation(), layer_attr=ExtraAttr(drop_rate=0.5)) - fc2 = fc_layer(input=bn, size=512, act=LinearActivation()) - return fc2 - - ``` - - 2.1. 首先定义了一组卷积网络，即conv_block。卷积核大小为3x3，池化窗口大小为2x2，窗口滑动大小为2，groups决定每组VGG模块是几次连续的卷积操作，dropouts指定Dropout操作的概率。所使用的`img_conv_group`是在`paddle.trainer_config_helpers`中预定义的模块，由若干组 `Conv->BN->ReLu->Dropout` 和 一组 `Pooling` 组成， - - 2.2. 五组卷积操作，即 5个conv_block。 第一、二组采用两次连续的卷积操作。第三、四、五组采用三次连续的卷积操作。每组最后一个卷积后面Dropout概率为0，即不使用Dropout操作。 - - 2.3. 最后接两层512维的全连接。 + ```python + net = vgg_bn_drop(data) + ``` + VGG核心模块的输入是数据层，`vgg_bn_drop` 定义了16层VGG结构，每层卷积后面引入BN层和Dropout层，详细的定义如下： + + ```python + def vgg_bn_drop(input, num_channels): + def conv_block(ipt, num_filter, groups, dropouts, num_channels_=None): + return img_conv_group( + input=ipt, + num_channels=num_channels_, + pool_size=2, + pool_stride=2, + conv_num_filter=[num_filter] * groups, + conv_filter_size=3, + conv_act=ReluActivation(), + conv_with_batchnorm=True, + conv_batchnorm_drop_rate=dropouts, + pool_type=MaxPooling()) + + conv1 = conv_block(input, 64, 2, [0.3, 0], 3) + conv2 = conv_block(conv1, 128, 2, [0.4, 0]) + conv3 = conv_block(conv2, 256, 3, [0.4, 0.4, 0]) + conv4 = conv_block(conv3, 512, 3, [0.4, 0.4, 0]) + conv5 = conv_block(conv4, 512, 3, [0.4, 0.4, 0]) + + drop = dropout_layer(input=conv5, dropout_rate=0.5) + fc1 = fc_layer(input=drop, size=512, act=LinearActivation()) + bn = batch_norm_layer( + input=fc1, act=ReluActivation(), layer_attr=ExtraAttr(drop_rate=0.5)) + fc2 = fc_layer(input=bn, size=512, act=LinearActivation()) + return fc2 + + ``` + + 2.1. 首先定义了一组卷积网络，即conv_block。卷积核大小为3x3，池化窗口大小为2x2，窗口滑动大小为2，groups决定每组VGG模块是几次连续的卷积操作，dropouts指定Dropout操作的概率。所使用的`img_conv_group`是在`paddle.trainer_config_helpers`中预定义的模块，由若干组 `Conv->BN->ReLu->Dropout` 和 一组 `Pooling` 组成， + + 2.2. 五组卷积操作，即 5个conv_block。 第一、二组采用两次连续的卷积操作。第三、四、五组采用三次连续的卷积操作。每组最后一个卷积后面Dropout概率为0，即不使用Dropout操作。 + + 2.3. 最后接两层512维的全连接。 3. 定义分类器 - 通过上面VGG网络提取高层特征，然后经过全连接层映射到类别维度大小的向量，再通过Softmax归一化得到每个类别的概率，也可称作分类器。 + 通过上面VGG网络提取高层特征，然后经过全连接层映射到类别维度大小的向量，再通过Softmax归一化得到每个类别的概率，也可称作分类器。 - ```python - out = fc_layer(input=net, size=class_num, act=SoftmaxActivation()) - ``` + ```python + out = fc_layer(input=net, size=class_num, act=SoftmaxActivation()) + ``` 4. 定义损失函数和网络输出 - 在有监督训练中需要输入图像对应的类别信息，同样通过`data_layer`来定义。训练中采用多类交叉熵作为损失函数，并作为网络的输出，预测阶段定义网络的输出为分类器得到的概率信息。 + 在有监督训练中需要输入图像对应的类别信息，同样通过`data_layer`来定义。训练中采用多类交叉熵作为损失函数，并作为网络的输出，预测阶段定义网络的输出为分类器得到的概率信息。 - ```python - if not is_predict: - lbl = data_layer(name="label", size=class_num) - cost = classification_cost(input=out, label=lbl) - outputs(cost) - else: - outputs(out) - ``` + ```python + if not is_predict: + lbl = data_layer(name="label", size=class_num) + cost = classification_cost(input=out, label=lbl) + outputs(cost) + else: + outputs(out) + ``` ### ResNet diff --git a/image_classification/deprecated/classify.py b/image_classification/deprecated/classify.py index 5a49bc22b0b205f7212c52c482f26720fea4e684..5b6e8a48d4dc72d7e3c69b6b2f46b3d41caee012 100644 --- a/image_classification/deprecated/classify.py +++ b/image_classification/deprecated/classify.py @@ -44,8 +44,9 @@ def vis_square(data, fname): (0, 1)) # add some space between filters + ((0, 0), ) * (data.ndim - 3)) # don't pad the last dimension (if there is one) - data = np.pad(data, padding, mode='constant', - constant_values=1) # pad with ones (white) + data = np.pad( + data, padding, mode='constant', + constant_values=1) # pad with ones (white) # tile the filters into an image data = data.reshape((n, n) + data.shape[1:]).transpose((0, 2, 1, 3) + tuple( range(4, data.ndim + 1))) diff --git a/image_classification/index.en.html b/image_classification/index.en.html index 23775756ba684905e585f5b9dd682015b1ad7fda..b26f8724298ac8b3bbb99896beb9b47f05d861ee 100644 --- a/image_classification/index.en.html +++ b/image_classification/index.en.html @@ -43,7 +43,7 @@ Image Classification ======================= -The source code of this chapter is in [book/image_classification](https://github.com/PaddlePaddle/book/tree/develop/image_classification). For the first-time users, please refer to PaddlePaddle[Installation Tutorial](http://www.paddlepaddle.org/doc_cn/build_and_install/index.html) for installation instructions. +The source code of this chapter is in [book/image_classification](https://github.com/PaddlePaddle/book/tree/develop/image_classification). For the first-time users, please refer to PaddlePaddle [Installation Tutorial](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/getstarted/build_and_install/docker_install_en.rst) for installation instructions. ## Background @@ -177,146 +177,73 @@ Figure 10. ResNet model for ImageNet

-## Data Preparation - -### Data description and downloading +## Dataset Commonly used public datasets for image classification are CIFAR(https://www.cs.toronto.edu/~kriz/cifar.html), ImageNet(http://image-net.org/), COCO(http://mscoco.org/), etc. Those used for fine-grained image classification are CUB-200-2011(http://www.vision.caltech.edu/visipedia/CUB-200-2011.html), Stanford Dog(http://vision.stanford.edu/aditya86/ImageNetDogs/), Oxford-flowers(http://www.robots.ox.ac.uk/~vgg/data/flowers/), etc. Among them, ImageNet are the largest and most research results are reported on ImageNet as mentioned in Model Overview section. Since 2010, the data of Imagenet has gone through some changes. The commonly used ImageNet-2012 dataset contains 1000 categories. There are 1,281,167 training images, ranging from 732 to 1200 images per category, and 50,000 validation images with 50 images per category in average. -Since ImageNet is too large to be downloaded and trained efficiently, we use CIFAR10 (https://www.cs.toronto.edu/~kriz/cifar.html) in this tutorial. The CIFAR-10 dataset consists of 60000 32x32 color images in 10 classes, with 6000 images per class. There are 50000 training images and 10000 test images. Figure 11 shows all the classes in CIFAR10 as well as 10 images randomly sampled from each category. +Since ImageNet is too large to be downloaded and trained efficiently, we use CIFAR-10 (https://www.cs.toronto.edu/~kriz/cifar.html) in this tutorial. The CIFAR-10 dataset consists of 60000 32x32 color images in 10 classes, with 6000 images per class. There are 50000 training images and 10000 test images. Figure 11 shows all the classes in CIFAR-10 as well as 10 images randomly sampled from each category.


Figure 11. CIFAR10 dataset[21]

-The following command is used for downloading data and calculating the mean image used for data preprocessing. - -```bash -./data/get_data.sh -``` + `paddle.datasets` package encapsulates multiple public datasets, including `cifar`, `imdb`, `mnist`, `moivelens` and `wmt14`, etc. There's no need for us to manually download and preprocess CIFAR-10. -### Data provider for PaddlePaddle +After issuing a command `python train.py`, training will starting immediately. The details will be unpacked by the following sessions to see how it works. -We use Python interface for providing data to PaddlePaddle. The following file dataprovider.py is a complete example for CIFAR10. +## Model Structure -- 'initializer' function performs initialization of dataprovider: loading the mean image, defining two input types -- image and label. +### Initialize PaddlePaddle -- 'process' function sends preprocessed data to PaddlePaddle. Data preprocessing performed in this function includes data perturbation, random horizontal flipping, deducting mean image from the raw image. +We must import and initialize PaddlePaddle (enable/disable GPU, set the number of trainers, etc). ```python -import numpy as np -import cPickle -from paddle.trainer.PyDataProvider2 import * - -def initializer(settings, mean_path, is_train, **kwargs): - settings.is_train = is_train - settings.input_size = 3 * 32 * 32 - settings.mean = np.load(mean_path)['mean'] - settings.input_types = { - 'image': dense_vector(settings.input_size), - 'label': integer_value(10) - } - - -@provider(init_hook=initializer, pool_size=50000) -def process(settings, file_list): - with open(file_list, 'r') as fdata: - for fname in fdata: - fo = open(fname.strip(), 'rb') - batch = cPickle.load(fo) - fo.close() - images = batch['data'] - labels = batch['labels'] - for im, lab in zip(images, labels): - if settings.is_train and np.random.randint(2): - im = im.reshape(3, 32, 32) - im = im[:,:,::-1] - im = im.flatten() - im = im - settings.mean - yield { - 'image': im.astype('float32'), - 'label': int(lab) - } -``` +import sys +import paddle.v2 as paddle -## Model Config - -### Data Definition - -In model config file, function `define_py_data_sources2` sets argument 'module' to dataprovider file for loading data, 'args' to mean image file. If the config file is used for prediction, then there is no need to set argument 'train_list'. - -```python -from paddle.trainer_config_helpers import * - -is_predict = get_config_arg("is_predict", bool, False) -if not is_predict: - define_py_data_sources2( - train_list='data/train.list', - test_list='data/test.list', - module='dataprovider', - obj='process', - args={'mean_path': 'data/mean.meta'}) -``` - -### Algorithm Settings - -In model config file, function 'settings' specifies optimization algorithm, batch size, learning rate, momentum and L2 regularization. - -```python -settings( - batch_size=128, - learning_rate=0.1 / 128.0, - learning_rate_decay_a=0.1, - learning_rate_decay_b=50000 * 100, - learning_rate_schedule='discexp', - learning_method=MomentumOptimizer(0.9), - regularization=L2Regularization(0.0005 * 128),) +# PaddlePaddle init +paddle.init(use_gpu=False, trainer_count=1) ``` -The learning rate adjustment policy can be defined with variables `learning_rate_decay_a`($a$), `learning_rate_decay_b`($b$) and `learning_rate_schedule`. In this example, discrete exponential method is used for adjusting learning rate. The formula is as follows, -$$ lr = lr_{0} * a^ {\lfloor \frac{n}{ b}\rfloor} $$ -where $n$ is the number of processed samples, $lr_{0}$ is the learning_rate set in 'settings'. - -### Model Architecture - -Here we provide the cofig files for VGG and ResNet models. +As alluded to in section [Model Overview](#model-overview), here we provide the implementations of both VGG and ResNet models. -#### VGG +### VGG -First we define VGG network. Since the image size and amount of CIFAR10 are relatively small comparing to ImageNet, we uses a small version of VGG network for CIFAR10. Convolution groups incorporate BN and dropout operations. +First, we use a VGG network. Since the image size and amount of CIFAR10 are relatively small comparing to ImageNet, we uses a small version of VGG network for CIFAR10. Convolution groups incorporate BN and dropout operations. 1. Define input data and its dimension - The input to the network is defined as `data_layer`, or image pixels in the context of image classification. The images in CIFAR10 are 32x32 color images of three channels. Therefore, the size of the input data is 3072 (3x32x32), and the number of categories is 10. + The input to the network is defined as `paddle.layer.data`, or image pixels in the context of image classification. The images in CIFAR10 are 32x32 color images of three channels. Therefore, the size of the input data is 3072 (3x32x32), and the number of categories is 10. ```python datadim = 3 * 32 * 32 classdim = 10 - data = data_layer(name='image', size=datadim) + image = paddle.layer.data( + name="image", type=paddle.data_type.dense_vector(datadim)) ``` 2. Define VGG main module ```python - net = vgg_bn_drop(data) + net = vgg_bn_drop(image) ``` - The input to VGG main module is from data layer. `vgg_bn_drop` defines a 16-layer VGG network, with each convolutional layer followed by BN and dropout layers. Here is the definition in detail: + The input to VGG main module is from the data layer. `vgg_bn_drop` defines a 16-layer VGG network, with each convolutional layer followed by BN and dropout layers. Here is the definition in detail: ```python - def vgg_bn_drop(input, num_channels): - def conv_block(ipt, num_filter, groups, dropouts, num_channels_=None): - return img_conv_group( + def vgg_bn_drop(input): + def conv_block(ipt, num_filter, groups, dropouts, num_channels=None): + return paddle.networks.img_conv_group( input=ipt, - num_channels=num_channels_, + num_channels=num_channels, pool_size=2, pool_stride=2, conv_num_filter=[num_filter] * groups, conv_filter_size=3, - conv_act=ReluActivation(), + conv_act=paddle.activation.Relu(), conv_with_batchnorm=True, conv_batchnorm_drop_rate=dropouts, - pool_type=MaxPooling()) + pool_type=paddle.pooling.Max()) conv1 = conv_block(input, 64, 2, [0.3, 0], 3) conv2 = conv_block(conv1, 128, 2, [0.4, 0]) @@ -324,16 +251,17 @@ First we define VGG network. Since the image size and amount of CIFAR10 are rela conv4 = conv_block(conv3, 512, 3, [0.4, 0.4, 0]) conv5 = conv_block(conv4, 512, 3, [0.4, 0.4, 0]) - drop = dropout_layer(input=conv5, dropout_rate=0.5) - fc1 = fc_layer(input=drop, size=512, act=LinearActivation()) - bn = batch_norm_layer( - input=fc1, act=ReluActivation(), layer_attr=ExtraAttr(drop_rate=0.5)) - fc2 = fc_layer(input=bn, size=512, act=LinearActivation()) + drop = paddle.layer.dropout(input=conv5, dropout_rate=0.5) + fc1 = paddle.layer.fc(input=drop, size=512, act=paddle.activation.Linear()) + bn = paddle.layer.batch_norm( + input=fc1, + act=paddle.activation.Relu(), + layer_attr=paddle.attr.Extra(drop_rate=0.5)) + fc2 = paddle.layer.fc(input=bn, size=512, act=paddle.activation.Linear()) return fc2 - ``` - 2.1. First defines a convolution block or conv_block. The default convolution kernel is 3x3, and the default pooling size is 2x2 with stride 2. Dropout specifies the probability in dropout operation. Function `img_conv_group` is defined in `paddle.trainer_config_helpers` consisting of a series of `Conv->BN->ReLu->Dropout` and a `Pooling`. + 2.1. First defines a convolution block or conv_block. The default convolution kernel is 3x3, and the default pooling size is 2x2 with stride 2. Dropout specifies the probability in dropout operation. Function `img_conv_group` is defined in `paddle.networks` consisting of a series of `Conv->BN->ReLu->Dropout` and a `Pooling`. 2.2. Five groups of convolutions. The first two groups perform two convolutions, while the last three groups perform three convolutions. The dropout rate of the last convolution in each group is set to 0, which means there is no dropout for this layer. @@ -351,15 +279,12 @@ First we define VGG network. Since the image size and amount of CIFAR10 are rela 4. Define Loss Function and Outputs - In the context of supervised learning, labels of training images are defined in `data_layer`, too. During training, cross-entropy is used as loss function and as the output of the network; During testing, the outputs are the probabilities calculated in the classifier. + In the context of supervised learning, labels of training images are defined in `paddle.layer.data`, too. During training, cross-entropy is used as loss function and as the output of the network; During testing, the outputs are the probabilities calculated in the classifier. ```python - if not is_predict: - lbl = data_layer(name="label", size=class_num) - cost = classification_cost(input=out, label=lbl) - outputs(cost) - else: - outputs(out) + lbl = paddle.layer.data( + name="label", type=paddle.data_type.integer_value(classdim)) + cost = paddle.layer.classification_cost(input=out, label=lbl) ``` ### ResNet @@ -367,13 +292,13 @@ First we define VGG network. Since the image size and amount of CIFAR10 are rela The first, third and forth steps of a ResNet are the same as a VGG. The second one is the main module. ```python -net = resnet_cifar10(data, depth=56) +net = resnet_cifar10(data, depth=32) ``` Here are some basic functions used in `resnet_cifar10`: - `conv_bn_layer` : convolutional layer followed by BN. - - `shortcut` : the shortcut branch in a residual block. There are two kinds of shortcuts: 1x1 convolution used when the number of channels between input and output are different; direct connection used otherwise. + - `shortcut` : the shortcut branch in a residual block. There are two kinds of shortcuts: 1x1 convolution used when the number of channels between input and output is different; direct connection used otherwise. - `basicblock` : a basic residual module as shown in the left of Figure 9, consisting of two sequential 3x3 convolutions and one "shortcut" branch. - `bottleneck` : a bottleneck module as shown in the right of Figure 9, consisting of a two 1x1 convolutions with one 3x3 convolution in between branch and a "shortcut" branch. @@ -385,47 +310,38 @@ def conv_bn_layer(input, filter_size, stride, padding, - active_type=ReluActivation(), + active_type=paddle.activation.Relu(), ch_in=None): - tmp = img_conv_layer( + tmp = paddle.layer.img_conv( input=input, filter_size=filter_size, num_channels=ch_in, num_filters=ch_out, stride=stride, padding=padding, - act=LinearActivation(), + act=paddle.activation.Linear(), bias_attr=False) - return batch_norm_layer(input=tmp, act=active_type) - + return paddle.layer.batch_norm(input=tmp, act=active_type) def shortcut(ipt, n_in, n_out, stride): if n_in != n_out: - return conv_bn_layer(ipt, n_out, 1, stride, 0, LinearActivation()) + return conv_bn_layer(ipt, n_out, 1, stride, 0, + paddle.activation.Linear()) else: return ipt def basicblock(ipt, ch_out, stride): - ch_in = ipt.num_filters + ch_in = ch_out * 2 tmp = conv_bn_layer(ipt, ch_out, 3, stride, 1) - tmp = conv_bn_layer(tmp, ch_out, 3, 1, 1, LinearActivation()) - short = shortcut(ipt, ch_in, ch_out, stride) - return addto_layer(input=[ipt, short], act=ReluActivation()) - -def bottleneck(ipt, ch_out, stride): - ch_in = ipt.num_filter - tmp = conv_bn_layer(ipt, ch_out, 1, stride, 0) - tmp = conv_bn_layer(tmp, ch_out, 3, 1, 1) - tmp = conv_bn_layer(tmp, ch_out * 4, 1, 1, 0, LinearActivation()) + tmp = conv_bn_layer(tmp, ch_out, 3, 1, 1, paddle.activation.Linear()) short = shortcut(ipt, ch_in, ch_out, stride) - return addto_layer(input=[ipt, short], act=ReluActivation()) + return paddle.layer.addto(input=[tmp, short], act=paddle.activation.Relu()) def layer_warp(block_func, ipt, features, count, stride): tmp = block_func(ipt, features, stride) for i in range(1, count): tmp = block_func(tmp, features, 1) return tmp - ``` The following are the components of `resnet_cifar10`: @@ -437,106 +353,131 @@ The following are the components of `resnet_cifar10`: Note: besides the first convolutional layer and the last fully-connected layer, the total number of layers in three `layer_warp` should be dividable by 6, that is the depth of `resnet_cifar10` should satisfy $(depth - 2) % 6 == 0$. ```python -def resnet_cifar10(ipt, depth=56): +def resnet_cifar10(ipt, depth=32): # depth should be one of 20, 32, 44, 56, 110, 1202 assert (depth - 2) % 6 == 0 n = (depth - 2) / 6 nStages = {16, 64, 128} - conv1 = conv_bn_layer(ipt, - ch_in=3, - ch_out=16, - filter_size=3, - stride=1, - padding=1) + conv1 = conv_bn_layer( + ipt, ch_in=3, ch_out=16, filter_size=3, stride=1, padding=1) res1 = layer_warp(basicblock, conv1, 16, n, 1) res2 = layer_warp(basicblock, res1, 32, n, 2) res3 = layer_warp(basicblock, res2, 64, n, 2) - pool = img_pool_layer(input=res3, - pool_size=8, - stride=1, - pool_type=AvgPooling()) + pool = paddle.layer.img_pool( + input=res3, pool_size=8, stride=1, pool_type=paddle.pooling.Avg()) return pool ``` ## Model Training -We can train the model by running the script train.sh, which specifies config file, device type, number of threads, number of passes, path to the trained models, etc, +### Define Parameters -``` bash -sh train.sh -``` +First, we create the model parameters according to the previous model configuration `cost`. -Here is an example script `train.sh`: - -```bash -#cfg=models/resnet.py -cfg=models/vgg.py -output=output -log=train.log - -paddle train \ - --config=$cfg \ - --use_gpu=true \ - --trainer_count=1 \ - --log_period=100 \ - --num_passes=300 \ - --save_dir=$output \ - 2>&1 | tee $log +```python +# Create parameters +parameters = paddle.parameters.create(cost) ``` -- `--config=$cfg` : specifies config file. The default is `models/vgg.py`. -- `--use_gpu=true` : uses GPU for training. If use CPU，set it to be false. -- `--trainer_count=1` : specifies the number of threads or GPUs. -- `--log_period=100` : specifies the number of batches between two logs. -- `--save_dir=$output` : specifies the path for saving trained models. +### Create Trainer -Here is an example log after training for one pass. The average error rates are 0.79958 on training set and 0.7858 on validation set. +Before jumping into creating a training module, algorithm setting is also necessary. +Here we specified `Momentum` optimization algorithm via `paddle.optimizer`. -```text -TrainerInternal.cpp:165] Batch=300 samples=38400 AvgCost=2.07708 CurrentCost=1.96158 Eval: classification_error_evaluator=0.81151 CurrentEval: classification_error_evaluator=0.789297 -TrainerInternal.cpp:181] Pass=0 Batch=391 samples=50000 AvgCost=2.03348 Eval: classification_error_evaluator=0.79958 -Tester.cpp:115] Test samples=10000 cost=1.99246 Eval: classification_error_evaluator=0.7858 +```python +# Create optimizer +momentum_optimizer = paddle.optimizer.Momentum( + momentum=0.9, + regularization=paddle.optimizer.L2Regularization(rate=0.0002 * 128), + learning_rate=0.1 / 128.0, + learning_rate_decay_a=0.1, + learning_rate_decay_b=50000 * 100, + learning_rate_schedule='discexp', + batch_size=128) + +# Create trainer +trainer = paddle.trainer.SGD(cost=cost, + parameters=parameters, + update_equation=momentum_optimizer) ``` -Figure 12 shows the curve of training error rate, which indicates it converges at Pass 200 with error rate 8.54%. +The learning rate adjustment policy can be defined with variables `learning_rate_decay_a`($a$), `learning_rate_decay_b`($b$) and `learning_rate_schedule`. In this example, discrete exponential method is used for adjusting learning rate. The formula is as follows, +$$ lr = lr_{0} * a^ {\lfloor \frac{n}{ b}\rfloor} $$ +where $n$ is the number of processed samples, $lr_{0}$ is the learning_rate. -

-
-Figure 12. The error rate of VGG model on CIFAR10 -

+### Training -## Model Application +`cifar.train10()` will yield records during each pass, after shuffling, a batch input is generated for training. -After training is done, the model from each pass is saved in `output/pass-%05d`. For example, the model of Pass 300 is saved in `output/pass-00299`. The script `classify.py` can be used to extract features and to classify an image. The default config file of this script is `models/vgg.py`. +```python +reader=paddle.batch( + paddle.reader.shuffle( + paddle.dataset.cifar.train10(), buf_size=50000), + batch_size=128) +``` + +`feeding` is devoted to specifying the correspondence between each yield record and `paddle.layer.data`. For instance, + the first column of data generated by `cifar.train10()` corresponds to image layer's feature. +```python +feeding={'image': 0, + 'label': 1} +``` -### Prediction +Callback function `event_handler` will be called during training when a pre-defined event happens. -We can run the following script to predict the category of an image. The default device is GPU. If to use CPU, set `-c`. -```bash -python classify.py --job=predict --model=output/pass-00299 --data=image/dog.png # -c +```python +# event handler to track training and testing process +def event_handler(event): + if isinstance(event, paddle.event.EndIteration): + if event.batch_id % 100 == 0: + print "\nPass %d, Batch %d, Cost %f, %s" % ( + event.pass_id, event.batch_id, event.cost, event.metrics) + else: + sys.stdout.write('.') + sys.stdout.flush() + if isinstance(event, paddle.event.EndPass): + result = trainer.test( + reader=paddle.batch( + paddle.dataset.cifar.test10(), batch_size=128), + feeding=feeding) + print "\nTest with Pass %d, %s" % (event.pass_id, result.metrics) ``` -Here is the result： +Finally, we can invoke `trainer.train` to start training: -```text -Label of image/dog.png is: 5 +```python +trainer.train( + reader=reader, + num_passes=200, + event_handler=event_handler, + feeding=feeding) ``` -### Feature Extraction +Here is an example log after training for one pass. The average error rates are 0.6875 on the training set and 0.8852 on the validation set. -We can run the following command to extract features from an image. Here `job` should be `extract` and the default layer is the first convolutional layer. Figure 13 shows the 64 feature maps output from the first convolutional layer of the VGG model. - -```bash -python classify.py --job=extract --model=output/pass-00299 --data=image/dog.png # -c +```text +Pass 0, Batch 0, Cost 2.473182, {'classification_error_evaluator': 0.9140625} +................................................................................................... +Pass 0, Batch 100, Cost 1.913076, {'classification_error_evaluator': 0.78125} +................................................................................................... +Pass 0, Batch 200, Cost 1.783041, {'classification_error_evaluator': 0.7421875} +................................................................................................... +Pass 0, Batch 300, Cost 1.668833, {'classification_error_evaluator': 0.6875} +.......................................................................................... +Test with Pass 0, {'classification_error_evaluator': 0.885200023651123} ``` +Figure 12 shows the curve of training error rate, which indicates it converges at Pass 200 with error rate 8.54%.

-
-Figre 13. Visualization of convolution layer feature maps +
+Figure 12. The error rate of VGG model on CIFAR10

+ +After training is done, the model from each pass is saved in `output/pass-%05d`. For example, the model of Pass 300 is saved in `output/pass-00299`. + ## Conclusion Traditional image classification methods involve multiple stages of processing and the framework is very complicated. In contrast, CNN models can be trained end-to-end with significant increase of classification accuracy. In this chapter, we introduce three models -- VGG, GoogleNet, ResNet, provide PaddlePaddle config files for training VGG and ResNet on CIFAR10, and explain how to perform prediction and feature extraction using PaddlePaddle API. For other datasets such as ImageNet, the procedure for config and training are the same and you are welcome to give it a try. @@ -589,7 +530,7 @@ Traditional image classification methods involve multiple stages of processing a [22] http://cs231n.github.io/classification/
-知识共享许可协议
本教程 由 PaddlePaddle 创作，采用 知识共享 署名-非商业性使用-相同方式共享 4.0 国际 许可协议进行许可。 +This tutorial is contributed by PaddlePaddle, and licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. diff --git a/image_classification/index.html b/image_classification/index.html index 97713196e8c9e40b21daea4dfd712661d15440a5..00efecac57fd0d375785a7d74c7c4c40791ce7d4 100644 --- a/image_classification/index.html +++ b/image_classification/index.html @@ -40,10 +40,9 @@ diff --git a/label_semantic_roles/index.html b/label_semantic_roles/index.html index 22aa1c174d4ad94ab0abf0f2a68707ed983b50f9..c90abc99fb28f25206d156e6cf1f3285731155f5 100644 --- a/label_semantic_roles/index.html +++ b/label_semantic_roles/index.html @@ -42,7 +42,7 @@