Skip to content

B
book

PaddlePaddle / book

通知 16
Star 4
Fork 0
B
book
提交 2f6da124 编写于 作者: G gongweibao 提交者： GitHub
浏览文件
操作

Merge pull request #150 from gongweibao/convertipynb 

convert markdown to ipynb file and check them
上级 4f1d0137 5f5910a9
隐藏空白更改
内联 并排
Showing with 2391 addition and 779 deletion
.pre-commit-config.yaml
浏览文件 @ 2f6da124
......@@ -32,3 +32,11 @@
entry: python pre-commit-hooks/convert_markdown_into_html.py
language: system
files: \.md$
- repo: local
hooks:
- id: convert-markdown-into-ipynb
name: convert-markdown-into-ipynb
description: Convert README.md into README.ipynb and README.en.md into README.en.ipynb
entry: ./pre-commit-hooks/convert_markdown_into_ipynb.sh
language: system
files: \.md$
.travis.yml
浏览文件 @ 2f6da124
......@@ -14,8 +14,10 @@ addons:
- python
- python-pip
- python2.7-dev
- golang
before_install:
- pip install virtualenv pre-commit
- GOPATH=/tmp/go go get -u github.com/wangkuiyi/ipynb/markdown-to-ipynb
script:
- travis/precommit.sh
notifications:
......
fit_a_line/README.en.ipynb 0 → 100644
浏览文件 @ 2f6da124
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Linear Regression\n",
"Let us begin the tutorial with a classical problem called Linear Regression \\[[1](#References)\\]. In this chapter, we will train a model from a realistic dataset to predict home prices. Some important concepts in Machine Learning will be covered through this example.\n",
"\n",
"The source code for this tutorial lives on [book/fit_a_line](https://github.com/PaddlePaddle/book/tree/develop/fit_a_line). For instructions on getting started with PaddlePaddle, see [PaddlePaddle installation guide](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/getstarted/build_and_install/docker_install_en.rst).\n",
"\n",
"## Problem Setup\n",
"Suppose we have a dataset of $n$ real estate properties. These real estate properties will be referred to as *homes* in this chapter for clarity.\n",
"\n",
"Each home is associated with $d$ attributes. The attributes describe characteristics such the number of rooms in the home, the number of schools or hospitals in the neighborhood, and the traffic condition nearby.\n",
"\n",
"In our problem setup, the attribute $x_{i,j}$ denotes the $j$th characteristic of the $i$th home. In addition, $y_i$ denotes the price of the $i$th home. Our task is to predict $y_i$ given a set of attributes $\\{x_{i,1}, ..., x_{i,d}\\}$. We assume that the price of a home is a linear combination of all of its attributes, namely,\n",
"\n",
"$$y_i = \\omega_1x_{i,1} + \\omega_2x_{i,2} + \\ldots + \\omega_dx_{i,d} + b, i=1,\\ldots,n$$\n",
"\n",
"where $\\vec{\\omega}$ and $b$ are the model parameters we want to estimate. Once they are learned, we will be able to predict the price of a home, given the attributes associated with it. We call this model **Linear Regression**. In other words, we want to regress a value against several values linearly. In practice, a linear model is often too simplistic to capture the real relationships between the variables. Yet, because Linear Regression is easy to train and analyze, it has been applied to a large number of real problems. As a result, it is an important topic in many classic Statistical Learning and Machine Learning textbooks \\[[2,3,4](#References)\\].\n",
"\n",
"## Results Demonstration\n",
"We first show the result of our model. The dataset [UCI Housing Data Set](https://archive.ics.uci.edu/ml/datasets/Housing) is used to train a linear model to predict the home prices in Boston. The figure below shows the predictions the model makes for some home prices. The $X$-axis represents the median value of the prices of simlilar homes within a bin, while the $Y$-axis represents the home value our linear model predicts. The dotted line represents points where $X=Y$. When reading the diagram, the more precise the model predicts, the closer the point is to the dotted line.\n",
"\u003cp align=\"center\"\u003e\n",
" \u003cimg src = \"image/predictions_en.png\" width=400\u003e\u003cbr/\u003e\n",
" Figure 1. Predicted Value V.S. Actual Value\n",
"\u003c/p\u003e\n",
"\n",
"## Model Overview\n",
"\n",
"### Model Definition\n",
"\n",
"In the UCI Housing Data Set, there are 13 home attributes $\\{x_{i,j}\\}$ that are related to the median home price $y_i$, which we aim to predict. Thus, our model can be written as:\n",
"\n",
"$$\\hat{Y} = \\omega_1X_{1} + \\omega_2X_{2} + \\ldots + \\omega_{13}X_{13} + b$$\n",
"\n",
"where $\\hat{Y}$ is the predicted value used to differentiate from actual value $Y$. The model learns parameters $\\omega_1, \\ldots, \\omega_{13}, b$, where the entries of $\\vec{\\omega}$ are **weights** and $b$ is **bias**.\n",
"\n",
"Now we need an objective to optimize, so that the learned parameters can make $\\hat{Y}$ as close to $Y$ as possible. Let's refer to the concept of [Loss Function (Cost Function)](https://en.wikipedia.org/wiki/Loss_function). A loss function must output a non-negative value, given any pair of the actual value $y_i$ and the predicted value $\\hat{y_i}$. This value reflects the magnitutude of the model error.\n",
"\n",
"For Linear Regression, the most common loss function is [Mean Square Error (MSE)](https://en.wikipedia.org/wiki/Mean_squared_error) which has the following form:\n",
"\n",
"$$MSE=\\frac{1}{n}\\sum_{i=1}^{n}{(\\hat{Y_i}-Y_i)}^2$$\n",
"\n",
"That is, for a dataset of size $n$, MSE is the average value of the the prediction sqaure errors.\n",
"\n",
"### Training\n",
"\n",
"After setting up our model, there are several major steps to go through to train it:\n",
"1. Initialize the parameters including the weights $\\vec{\\omega}$ and the bias $b$. For example, we can set their mean values as $0$s, and their standard deviations as $1$s.\n",
"2. Feedforward. Evaluate the network output and compute the corresponding loss.\n",
"3. [Backpropagate](https://en.wikipedia.org/wiki/Backpropagation) the errors. The errors will be propagated from the output layer back to the input layer, during which the model parameters will be updated with the corresponding errors.\n",
"4. Repeat steps 2~3, until the loss is below a predefined threshold or the maximum number of repeats is reached.\n",
"\n",
"## Dataset\n",
"\n",
"### Python Dataset Modules\n",
"\n",
"Our program starts with importing necessary packages:\n",
"\n"
]
},
{
"cell_type": "code",
"metadata": {
"editable": true
},
"source": [
"import paddle.v2 as paddle\n",
"import paddle.v2.dataset.uci_housing as uci_housing\n"
],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n"
]
}
],
"execution_count": 1
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"We encapsulated the [UCI Housing Data Set](https://archive.ics.uci.edu/ml/datasets/Housing) in our Python module `uci_housing`. This module can\n",
"\n",
"1. download the dataset to `~/.cache/paddle/dataset/uci_housing/housing.data`, if not yet, and\n",
"2. [preprocesses](#preprocessing) the dataset.\n",
"\n",
"### An Introduction of the Dataset\n",
"\n",
"The UCI housing dataset has 506 instances. Each instance describes the attributes of a house in surburban Boston. The attributes are explained below:\n",
"\n",
"| Attribute Name | Characteristic | Data Type |\n",
"| ------| ------ | ------ |\n",
"| CRIM | per capita crime rate by town | Continuous|\n",
"| ZN | proportion of residential land zoned for lots over 25,000 sq.ft. | Continuous |\n",
"| INDUS | proportion of non-retail business acres per town | Continuous |\n",
"| CHAS | Charles River dummy variable | Discrete, 1 if tract bounds river; 0 otherwise|\n",
"| NOX | nitric oxides concentration (parts per 10 million) | Continuous |\n",
"| RM | average number of rooms per dwelling | Continuous |\n",
"| AGE | proportion of owner-occupied units built prior to 1940 | Continuous |\n",
"| DIS | weighted distances to five Boston employment centres | Continuous |\n",
"| RAD | index of accessibility to radial highways | Continuous |\n",
"| TAX | full-value property-tax rate per $10,000 | Continuous |\n",
"| PTRATIO | pupil-teacher ratio by town | Continuous |\n",
"| B | 1000(Bk - 0.63)^2 where Bk is the proportion of blacks by town | Continuous |\n",
"| LSTAT | % lower status of the population | Continuous |\n",
"| MEDV | Median value of owner-occupied homes in $1000's | Continuous |\n",
"\n",
"The last entry is the median home price.\n",
"\n",
"### Preprocessing\n",
"#### Continuous and Discrete Data\n",
"We define a feature vector of length 13 for each home, where each entry corresponds to an attribute. Our first observation is that, among the 13 dimensions, there are 12 continuous dimensions and 1 discrete dimension.\n",
"\n",
"Note that although a discrete value is also written as numeric values such as 0, 1, or 2, its meaning differs from a continuous value drastically. The linear difference between two discrete values has no meaning. For example, suppose $0$, $1$, and $2$ are used to represent colors *Red*, *Green*, and *Blue* respectively. Judging from the numeric representation of these colors, *Red* differs more from *Blue* than it does from *Green*. Yet in actuality, it is not true that extent to which the color *Blue* is different from *Red* is greater than the extent to which *Green* is different from *Red*. Therefore, when handling a discrete feature that has $d$ possible values, we usually convert it to $d$ new features where each feature takes a binary value, $0$ or $1$, indicating whether the original value is absent or present. Alternatively, the discrete features can be mapped onto a continuous multi-dimensional vector through an embedding table. For our problem here, because CHAS itself is a binary discrete value, we do not need to do any preprocessing.\n",
"\n",
"#### Feature Normalization\n",
"We also observe a huge difference among the value ranges of the 13 features (Figure 2). For instance, the values of feature *B* fall in $[0.32, 396.90]$, whereas those of feature *NOX* has a range of $[0.3850, 0.8170]$. An effective optimization would require data normalization. The goal of data normalization is to scale te values of each feature into roughly the same range, perhaps $[-0.5, 0.5]$. Here, we adopt a popular normalization technique where we substract the mean value from the feature value and divide the result by the width of the original range.\n",
"\n",
"There are at least three reasons for [Feature Normalization](https://en.wikipedia.org/wiki/Feature_scaling) (Feature Scaling):\n",
"- A value range that is too large or too small might cause floating number overflow or underflow during computation.\n",
"- Different value ranges might result in varying *importances* of different features to the model (at least in the beginning of the training process). This assumption about the data is often unreasonable, making the optimization difficult, which in turn results in increased training time.\n",
"- Many machine learning techniques or models (e.g., *L1/L2 regularization* and *Vector Space Model*) assumes that all the features have roughly zero means and their value ranges are similar.\n",
"\n",
"\u003cp align=\"center\"\u003e\n",
" \u003cimg src = \"image/ranges_en.png\" width=550\u003e\u003cbr/\u003e\n",
" Figure 2. The value ranges of the features\n",
"\u003c/p\u003e\n",
"\n",
"#### Prepare Training and Test Sets\n",
"We split the dataset in two, one for adjusting the model parameters, namely, for model training, and the other for model testing. The model error on the former is called the **training error**, and the error on the latter is called the **test error**. Our goal in training a model is to find the statistical dependency between the outputs and the inputs, so that we can predict new outputs given new inputs. As a result, the test error reflects the performance of the model better than the training error does. We consider two things when deciding the ratio of the training set to the test set: 1) More training data will decrease the variance of the parameter estimation, yielding more reliable models; 2) More test data will decrease the variance of the test error, yielding more reliable test errors. One standard split ratio is $8:2$.\n",
"\n",
"\n",
"When training complex models, we usually have one more split: the validation set. Complex models usually have [Hyperparameters](https://en.wikipedia.org/wiki/Hyperparameter_optimization) that need to be set before the training process, such as the number of layers in the network. Because hyperparameters are not part of the model parameters, they cannot be trained using the same loss function. Thus we will try several sets of hyperparameters to train several models and cross-validate them on the validation set to pick the best one; finally, the selected trained model is tested on the test set. Because our model is relatively simple, we will omit this validation process.\n",
"\n",
"\n",
"## Training\n",
"\n",
"`fit_a_line/trainer.py` demonstrates the training using [PaddlePaddle](http://paddlepaddle.org).\n",
"\n",
"### Initialize PaddlePaddle\n",
"\n"
]
},
{
"cell_type": "code",
"metadata": {
"editable": true
},
"source": [
"paddle.init(use_gpu=False, trainer_count=1)\n"
],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n"
]
}
],
"execution_count": 1
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### Model Configuration\n",
"\n",
"Logistic regression is essentially a fully-connected layer with linear activation:\n",
"\n"
]
},
{
"cell_type": "code",
"metadata": {
"editable": true
},
"source": [
"x = paddle.layer.data(name='x', type=paddle.data_type.dense_vector(13))\n",
"y_predict = paddle.layer.fc(input=x,\n",
" size=1,\n",
" act=paddle.activation.Linear())\n",
"y = paddle.layer.data(name='y', type=paddle.data_type.dense_vector(1))\n",
"cost = paddle.layer.regression_cost(input=y_predict, label=y)\n"
],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n"
]
}
],
"execution_count": 1
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Create Parameters\n",
"\n"
]
},
{
"cell_type": "code",
"metadata": {
"editable": true
},
"source": [
"parameters = paddle.parameters.create(cost)\n"
],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n"
]
}
],
"execution_count": 1
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### Create Trainer\n",
"\n"
]
},
{
"cell_type": "code",
"metadata": {
"editable": true
},
"source": [
"optimizer = paddle.optimizer.Momentum(momentum=0)\n",
"\n",
"trainer = paddle.trainer.SGD(cost=cost,\n",
" parameters=parameters,\n",
" update_equation=optimizer)\n"
],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n"
]
}
],
"execution_count": 1
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### Feeding Data\n",
"\n",
"PaddlePaddle provides the\n",
"[reader mechanism](https://github.com/PaddlePaddle/Paddle/tree/develop/doc/design/reader)\n",
"for loadinng training data. A reader may return multiple columns, and we need a Python dictionary to specify the mapping from column index to data layers.\n",
"\n"
]
},
{
"cell_type": "code",
"metadata": {
"editable": true
},
"source": [
"feeding={'x': 0, 'y': 1}\n"
],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n"
]
}
],
"execution_count": 1
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"Moreover, an event handler is provided to print the training progress:\n",
"\n"
]
},
{
"cell_type": "code",
"metadata": {
"editable": true
},
"source": [
"# event_handler to print training and testing info\n",
"def event_handler(event):\n",
" if isinstance(event, paddle.event.EndIteration):\n",
" if event.batch_id % 100 == 0:\n",
" print \"Pass %d, Batch %d, Cost %f\" % (\n",
" event.pass_id, event.batch_id, event.cost)\n",
"\n",
" if isinstance(event, paddle.event.EndPass):\n",
" result = trainer.test(\n",
" reader=paddle.batch(\n",
" uci_housing.test(), batch_size=2),\n",
" feeding=feeding)\n",
" print \"Test %d, Cost %f\" % (event.pass_id, result.cost)\n"
],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n"
]
}
],
"execution_count": 1
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### Start Training\n",
"\n"
]
},
{
"cell_type": "code",
"metadata": {
"editable": true
},
"source": [
"trainer.train(\n",
" reader=paddle.batch(\n",
" paddle.reader.shuffle(\n",
" uci_housing.train(), buf_size=500),\n",
" batch_size=2),\n",
" feeding=feeding,\n",
" event_handler=event_handler,\n",
" num_passes=30)\n"
],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n"
]
}
],
"execution_count": 1
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Summary\n",
"This chapter introduces *Linear Regression* and how to train and test this model with PaddlePaddle, using the UCI Housing Data Set. Because a large number of more complex models and techniques are derived from linear regression, it is important to understand its underlying theory and limitation.\n",
"\n",
"\n",
"## References\n",
"1. https://en.wikipedia.org/wiki/Linear_regression\n",
"2. Friedman J, Hastie T, Tibshirani R. The elements of statistical learning[M]. Springer, Berlin: Springer series in statistics, 2001.\n",
"3. Murphy K P. Machine learning: a probabilistic perspective[M]. MIT press, 2012.\n",
"4. Bishop C M. Pattern recognition[J]. Machine Learning, 2006, 128.\n",
"\n",
"\u003cbr/\u003e\n",
"This tutorial is contributed by \u003ca xmlns:cc=\"http://creativecommons.org/ns#\" href=\"http://book.paddlepaddle.org\" property=\"cc:attributionName\" rel=\"cc:attributionURL\"\u003ePaddlePaddle\u003c/a\u003e, and licensed under a \u003ca rel=\"license\" href=\"http://creativecommons.org/licenses/by-nc-sa/4.0/\"\u003eCreative Commons Attribution-NonCommercial-ShareAlike 4.0 International License\u003c/a\u003e.\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.0"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
fit_a_line/README.ipynb 0 → 100644
浏览文件 @ 2f6da124
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 线性回归\n",
"让我们从经典的线性回归（Linear Regression \\[[1](#参考文献)\\]）模型开始这份教程。在这一章里，你将使用真实的数据集建立起一个房价预测模型，并且了解到机器学习中的若干重要概念。\n",
"\n",
"本教程源代码目录在[book/fit_a_line](https://github.com/PaddlePaddle/book/tree/develop/fit_a_line)， 初次使用请参考PaddlePaddle[安装教程](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/getstarted/build_and_install/docker_install_cn.rst)。\n",
"\n",
"## 背景介绍\n",
"给定一个大小为$n$的数据集 ${\\{y_{i}, x_{i1}, ..., x_{id}\\}}_{i=1}^{n}$，其中$x_{i1}, \\ldots, x_{id}$是第$i$个样本$d$个属性上的取值，$y_i$是该样本待预测的目标。线性回归模型假设目标$y_i$可以被属性间的线性组合描述，即\n",
"\n",
"$$y_i = \\omega_1x_{i1} + \\omega_2x_{i2} + \\ldots + \\omega_dx_{id} + b, i=1,\\ldots,n$$\n",
"\n",
"例如，在我们将要建模的房价预测问题里，$x_{ij}$是描述房子$i$的各种属性（比如房间的个数、周围学校和医院的个数、交通状况等），而 $y_i$是房屋的价格。\n",
"\n",
"初看起来，这个假设实在过于简单了，变量间的真实关系很难是线性的。但由于线性回归模型有形式简单和易于建模分析的优点，它在实际问题中得到了大量的应用。很多经典的统计学习、机器学习书籍\\[[2,3,4](#参考文献)\\]也选择对线性模型独立成章重点讲解。\n",
"\n",
"## 效果展示\n",
"我们使用从[UCI Housing Data Set](https://archive.ics.uci.edu/ml/datasets/Housing)获得的波士顿房价数据集进行模型的训练和预测。下面的散点图展示了使用模型对部分房屋价格进行的预测。其中，每个点的横坐标表示同一类房屋真实价格的中位数，纵坐标表示线性回归模型根据特征预测的结果，当二者值完全相等的时候就会落在虚线上。所以模型预测得越准确，则点离虚线越近。\n",
"\u003cp align=\"center\"\u003e\n",
" \u003cimg src = \"image/predictions.png\" width=400\u003e\u003cbr/\u003e\n",
" 图1. 预测值 V.S. 真实值\n",
"\u003c/p\u003e\n",
"\n",
"## 模型概览\n",
"\n",
"### 模型定义\n",
"\n",
"在波士顿房价数据集中，和房屋相关的值共有14个：前13个用来描述房屋相关的各种信息，即模型中的 $x_i$；最后一个值为我们要预测的该类房屋价格的中位数，即模型中的 $y_i$。因此，我们的模型就可以表示成：\n",
"\n",
"$$\\hat{Y} = \\omega_1X_{1} + \\omega_2X_{2} + \\ldots + \\omega_{13}X_{13} + b$$\n",
"\n",
"$\\hat{Y}$ 表示模型的预测结果，用来和真实值$Y$区分。模型要学习的参数即：$\\omega_1, \\ldots, \\omega_{13}, b$。\n",
"\n",
"建立模型后，我们需要给模型一个优化目标，使得学到的参数能够让预测值$\\hat{Y}$尽可能地接近真实值$Y$。这里我们引入损失函数（[Loss Function](https://en.wikipedia.org/wiki/Loss_function)，或Cost Function）这个概念。 输入任意一个数据样本的目标值$y_{i}$和模型给出的预测值$\\hat{y_{i}}$，损失函数输出一个非负的实值。这个实质通常用来反映模型误差的大小。\n",
"\n",
"对于线性回归模型来讲，最常见的损失函数就是均方误差（Mean Squared Error， [MSE](https://en.wikipedia.org/wiki/Mean_squared_error)）了，它的形式是：\n",
"\n",
"$$MSE=\\frac{1}{n}\\sum_{i=1}^{n}{(\\hat{Y_i}-Y_i)}^2$$\n",
"\n",
"即对于一个大小为$n$的测试集，$MSE$是$n$个数据预测结果误差平方的均值。\n",
"\n",
"### 训练过程\n",
"\n",
"定义好模型结构之后，我们要通过以下几个步骤进行模型训练\n",
" 1. 初始化参数，其中包括权重$\\omega_i$和偏置$b$，对其进行初始化（如0均值，1方差）。\n",
" 2. 网络正向传播计算网络输出和损失函数。\n",
" 3. 根据损失函数进行反向误差传播 （[backpropagation](https://en.wikipedia.org/wiki/Backpropagation)），将网络误差从输出层依次向前传递, 并更新网络中的参数。\n",
" 4. 重复2~3步骤，直至网络训练误差达到规定的程度或训练轮次达到设定值。\n",
"\n",
"## 数据集\n",
"\n",
"### 数据集接口的封装\n",
"首先加载需要的包\n",
"\n"
]
},
{
"cell_type": "code",
"metadata": {
"editable": true
},
"source": [
"import paddle.v2 as paddle\n",
"import paddle.v2.dataset.uci_housing as uci_housing\n"
],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n"
]
}
],
"execution_count": 1
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"我们通过uci_housing模块引入了数据集合[UCI Housing Data Set](https://archive.ics.uci.edu/ml/datasets/Housing)\n",
"\n",
"其中，在uci_housing模块中封装了：\n",
"\n",
"1. 数据下载的过程。下载数据保存在~/.cache/paddle/dataset/uci_housing/housing.data。\n",
"2. [数据预处理](#数据预处理)的过程。\n",
"\n",
"\n",
"### 数据集介绍\n",
"这份数据集共506行，每行包含了波士顿郊区的一类房屋的相关信息及该类房屋价格的中位数。其各维属性的意义如下：\n",
"\n",
"| 属性名 | 解释 | 类型 |\n",
"| ------| ------ | ------ |\n",
"| CRIM | 该镇的人均犯罪率 | 连续值 |\n",
"| ZN | 占地面积超过25,000平方呎的住宅用地比例 | 连续值 |\n",
"| INDUS | 非零售商业用地比例 | 连续值 |\n",
"| CHAS | 是否邻近 Charles River | 离散值，1=邻近；0=不邻近 |\n",
"| NOX | 一氧化氮浓度 | 连续值 |\n",
"| RM | 每栋房屋的平均客房数 | 连续值 |\n",
"| AGE | 1940年之前建成的自用单位比例 | 连续值 |\n",
"| DIS | 到波士顿5个就业中心的加权距离 | 连续值 |\n",
"| RAD | 到径向公路的可达性指数 | 连续值 |\n",
"| TAX | 全值财产税率 | 连续值 |\n",
"| PTRATIO | 学生与教师的比例 | 连续值 |\n",
"| B | 1000(BK - 0.63)^2，其中BK为黑人占比 | 连续值 |\n",
"| LSTAT | 低收入人群占比 | 连续值 |\n",
"| MEDV | 同类房屋价格的中位数 | 连续值 |\n",
"\n",
"### 数据预处理\n",
"#### 连续值与离散值\n",
"观察一下数据，我们的第一个发现是：所有的13维属性中，有12维的连续值和1维的离散值（CHAS）。离散值虽然也常使用类似0、1、2这样的数字表示，但是其含义与连续值是不同的，因为这里的差值没有实际意义。例如，我们用0、1、2来分别表示红色、绿色和蓝色的话，我们并不能因此说“蓝色和红色”比“绿色和红色”的距离更远。所以通常对一个有$d$个可能取值的离散属性，我们会将它们转为$d$个取值为0或1的二值属性或者将每个可能取值映射为一个多维向量。不过就这里而言，因为CHAS本身就是一个二值属性，就省去了这个麻烦。\n",
"\n",
"#### 属性的归一化\n",
"另外一个稍加观察即可发现的事实是，各维属性的取值范围差别很大（如图2所示）。例如，属性B的取值范围是[0.32, 396.90]，而属性NOX的取值范围是[0.3850, 0.8170]。这里就要用到一个常见的操作-归一化（normalization）了。归一化的目标是把各位属性的取值范围放缩到差不多的区间，例如[-0.5,0.5]。这里我们使用一种很常见的操作方法：减掉均值，然后除以原取值范围。\n",
"\n",
"做归一化（或 [Feature scaling](https://en.wikipedia.org/wiki/Feature_scaling)）至少有以下3个理由：\n",
"- 过大或过小的数值范围会导致计算时的浮点上溢或下溢。\n",
"- 不同的数值范围会导致不同属性对模型的重要性不同（至少在训练的初始阶段如此），而这个隐含的假设常常是不合理的。这会对优化的过程造成困难，使训练时间大大的加长。\n",
"- 很多的机器学习技巧/模型（例如L1，L2正则项，向量空间模型-Vector Space Model）都基于这样的假设：所有的属性取值都差不多是以0为均值且取值范围相近的。\n",
"\n",
"\u003cp align=\"center\"\u003e\n",
" \u003cimg src = \"image/ranges.png\" width=550\u003e\u003cbr/\u003e\n",
" 图2. 各维属性的取值范围\n",
"\u003c/p\u003e\n",
"\n",
"#### 整理训练集与测试集\n",
"我们将数据集分割为两份：一份用于调整模型的参数，即进行模型的训练，模型在这份数据集上的误差被称为**训练误差**；另外一份被用来测试，模型在这份数据集上的误差被称为**测试误差**。我们训练模型的目的是为了通过从训练数据中找到规律来预测未知的新数据，所以测试误差是更能反映模型表现的指标。分割数据的比例要考虑到两个因素：更多的训练数据会降低参数估计的方差，从而得到更可信的模型；而更多的测试数据会降低测试误差的方差，从而得到更可信的测试误差。我们这个例子中设置的分割比例为$8:2$\n",
"\n",
"\n",
"在更复杂的模型训练过程中，我们往往还会多使用一种数据集：验证集。因为复杂的模型中常常还有一些超参数（[Hyperparameter](https://en.wikipedia.org/wiki/Hyperparameter_optimization)）需要调节，所以我们会尝试多种超参数的组合来分别训练多个模型，然后对比它们在验证集上的表现选择相对最好的一组超参数，最后才使用这组参数下训练的模型在测试集上评估测试误差。由于本章训练的模型比较简单，我们暂且忽略掉这个过程。\n",
"\n",
"## 训练\n",
"\n",
"`fit_a_line/trainer.py`演示了训练的整体过程。\n",
"\n",
"### 初始化PaddlePaddle\n",
"\n"
]
},
{
"cell_type": "code",
"metadata": {
"editable": true
},
"source": [
"paddle.init(use_gpu=False, trainer_count=1)\n"
],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n"
]
}
],
"execution_count": 1
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### 模型配置\n",
"\n",
"线性回归的模型其实就是一个采用线性激活函数（linear activation，`LinearActivation`）的全连接层（fully-connected layer，`fc_layer`）：\n",
"\n"
]
},
{
"cell_type": "code",
"metadata": {
"editable": true
},
"source": [
"x = paddle.layer.data(name='x', type=paddle.data_type.dense_vector(13))\n",
"y_predict = paddle.layer.fc(input=x,\n",
" size=1,\n",
" act=paddle.activation.Linear())\n",
"y = paddle.layer.data(name='y', type=paddle.data_type.dense_vector(1))\n",
"cost = paddle.layer.regression_cost(input=y_predict, label=y)\n"
],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n"
]
}
],
"execution_count": 1
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 创建参数\n",
"\n"
]
},
{
"cell_type": "code",
"metadata": {
"editable": true
},
"source": [
"parameters = paddle.parameters.create(cost)\n"
],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n"
]
}
],
"execution_count": 1
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### 创建Trainer\n",
"\n"
]
},
{
"cell_type": "code",
"metadata": {
"editable": true
},
"source": [
"optimizer = paddle.optimizer.Momentum(momentum=0)\n",
"\n",
"trainer = paddle.trainer.SGD(cost=cost,\n",
" parameters=parameters,\n",
" update_equation=optimizer)\n"
],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n"
]
}
],
"execution_count": 1
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### 读取数据且打印训练的中间信息\n",
"\n",
"PaddlePaddle提供一个\n",
"[reader机制](https://github.com/PaddlePaddle/Paddle/tree/develop/doc/design/reader)\n",
"来读取数据。 Reader返回的数据可以包括多列，我们需要一个Python dict把列\n",
"序号映射到网络里的数据层。\n",
"\n"
]
},
{
"cell_type": "code",
"metadata": {
"editable": true
},
"source": [
"feeding={'x': 0, 'y': 1}\n"
],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n"
]
}
],
"execution_count": 1
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"此外，我们还可以提供一个 event handler，来打印训练的进度：\n",
"\n"
]
},
{
"cell_type": "code",
"metadata": {
"editable": true
},
"source": [
"# event_handler to print training and testing info\n",
"def event_handler(event):\n",
" if isinstance(event, paddle.event.EndIteration):\n",
" if event.batch_id % 100 == 0:\n",
" print \"Pass %d, Batch %d, Cost %f\" % (\n",
" event.pass_id, event.batch_id, event.cost)\n",
"\n",
" if isinstance(event, paddle.event.EndPass):\n",
" result = trainer.test(\n",
" reader=paddle.batch(\n",
" uci_housing.test(), batch_size=2),\n",
" feeding=feeding)\n",
" print \"Test %d, Cost %f\" % (event.pass_id, result.cost)\n"
],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n"
]
}
],
"execution_count": 1
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### 开始训练\n",
"\n"
]
},
{
"cell_type": "code",
"metadata": {
"editable": true
},
"source": [
"trainer.train(\n",
" reader=paddle.batch(\n",
" paddle.reader.shuffle(\n",
" uci_housing.train(), buf_size=500),\n",
" batch_size=2),\n",
" feeding=feeding,\n",
" event_handler=event_handler,\n",
" num_passes=30)\n"
],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n"
]
}
],
"execution_count": 1
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## 总结\n",
"在这章里，我们借助波士顿房价这一数据集，介绍了线性回归模型的基本概念，以及如何使用PaddlePaddle实现训练和测试的过程。很多的模型和技巧都是从简单的线性回归模型演化而来，因此弄清楚线性模型的原理和局限非常重要。\n",
"\n",
"\n",
"## 参考文献\n",
"1. https://en.wikipedia.org/wiki/Linear_regression\n",
"2. Friedman J, Hastie T, Tibshirani R. The elements of statistical learning[M]. Springer, Berlin: Springer series in statistics, 2001.\n",
"3. Murphy K P. Machine learning: a probabilistic perspective[M]. MIT press, 2012.\n",
"4. Bishop C M. Pattern recognition[J]. Machine Learning, 2006, 128.\n",
"\n",
"\u003cbr/\u003e\n",
"\u003ca rel=\"license\" href=\"http://creativecommons.org/licenses/by-nc-sa/4.0/\"\u003e\u003cimg alt=\"知识共享许可协议\" style=\"border-width:0\" src=\"https://i.creativecommons.org/l/by-nc-sa/4.0/88x31.png\" /\u003e\u003c/a\u003e\u003cbr /\u003e\u003cspan xmlns:dct=\"http://purl.org/dc/terms/\" href=\"http://purl.org/dc/dcmitype/Text\" property=\"dct:title\" rel=\"dct:type\"\u003e本教程\u003c/span\u003e 由 \u003ca xmlns:cc=\"http://creativecommons.org/ns#\" href=\"http://book.paddlepaddle.org\" property=\"cc:attributionName\" rel=\"cc:attributionURL\"\u003ePaddlePaddle\u003c/a\u003e 创作，采用 \u003ca rel=\"license\" href=\"http://creativecommons.org/licenses/by-nc-sa/4.0/\"\u003e知识共享 署名-非商业性使用-相同方式共享 4.0 国际 许可协议\u003c/a\u003e进行许可。\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.0"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
pre-commit-hooks/convert_markdown_into_ipynb.sh 0 → 100755
浏览文件 @ 2f6da124
#!/bin/sh
for file in $@ ; do
/tmp/go/bin/markdown-to-ipynb < $file > ${file%.*}".ipynb"
if [ $? -ne 0 ]; then
echo >&2 "markdown-to-ipynb $file error"
exit 1
fi
done
recommender_system/README.en.ipynb 0 → 100644
浏览文件 @ 2f6da124
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Personalized Recommendation\n",
"\n",
"The source code of this tutorial is in [book/recommender_system](https://github.com/PaddlePaddle/book/tree/develop/recommender_system).\n",
"\n",
"For instructions on getting started with PaddlePaddle, see [PaddlePaddle installation guide](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/getstarted/build_and_install/docker_install_en.rst).\n",
"\n",
"\n",
"## Background\n",
"\n",
"With the fast growth of e-commerce, online videos, and online reading business, users have to rely on recommender systems to avoid manually browsing tremendous volume of choices. Recommender systems understand users' interest by mining user behavior and other properties of users and products.\n",
"\n",
"Some well know approaches include:\n",
"\n",
"- User behavior-based approach. A well-known method is collaborative filtering. The underlying assumption is that if a person A has the same opinion as a person B on an issue, A is more likely to have B's opinion on a different issue than that of a randomly chosen person.\n",
"\n",
"- Content-based recommendation[[1](#reference)]. This approach infers feature vectors that represent products from their descriptions. It also infers feature vectors that represent users' interests. Then it measures the relevance of users and products by some distances between these feature vectors.\n",
"\n",
"- Hybrid approach[[2](#reference)]: This approach uses the content-based information to help address the cold start problem[[6](#reference)] in behavior-based approach.\n",
"\n",
"Among these options, collaborative filtering might be the most studied one. Some of its variants include user-based[[3](#reference)], item-based [[4](#reference)], social network based[[5](#reference)], and model-based.\n",
"\n",
"This tutorial explains a deep learning based approach and how to implement it using PaddlePaddle. We will train a model using a dataset that includes user information, movie information, and ratings. Once we train the model, we will be able to get a predicted rating given a pair of user and movie IDs.\n",
"\n",
"\n",
"## Model Overview\n",
"\n",
"To know more about deep learning based recommendation, let us start from going over the Youtube recommender system[[7](#参考文献)] before introducing our hybrid model.\n",
"\n",
"\n",
"### YouTube's Deep Learning Recommendation Model\n",
"\n",
"YouTube is a video-sharing Web site with one of the largest user base in the world. Its recommender system serves more than a billion users. This system is composed of two major parts: candidate generation and ranking. The former selects few hundreds of candidates from millions of videos, and the latter ranks and outputs the top 10.\n",
"\n",
"\u003cp align=\"center\"\u003e\n",
"\u003cimg src=\"image/YouTube_Overview.en.png\" width=\"70%\" \u003e\u003cbr/\u003e\n",
"Figure 1. YouTube recommender system overview.\n",
"\u003c/p\u003e\n",
"\n",
"#### Candidate Generation Network\n",
"\n",
"Youtube models candidate generation as a multiclass classification problem with a huge number of classes equal to the number of videos. The architecture of the model is as follows:\n",
"\n",
"\u003cp align=\"center\"\u003e\n",
"\u003cimg src=\"image/Deep_candidate_generation_model_architecture.en.png\" width=\"70%\" \u003e\u003cbr/\u003e\n",
"Figure. Deep candidate geeration model.\n",
"\u003c/p\u003e\n",
"\n",
"The first stage of this model maps watching history and search queries into fixed-length representative features. Then, an MLP (multi-layer perceptron, as described in the [Recognize Digits](https://github.com/PaddlePaddle/book/blob/develop/recognize_digits/README.md) tutorial) takes the concatenation of all representative vectors. The output of the MLP represents the user' *intrinsic interests*. At training time, it is used together with a softmax output layer for minimizing the classification error. At serving time, it is used to compute the relevance of the user with all movies.\n",
"\n",
"For a user $U$, the predicted watching probability of video $i$ is\n",
"\n",
"$$P(\\omega=i|u)=\\frac{e^{v_{i}u}}{\\sum_{j \\in V}e^{v_{j}u}}$$\n",
"\n",
"where $u$ is the representative vector of user $U$, $V$ is the corpus of all videos, $v_i$ is the representative vector of the $i$-th video. $u$ and $v_i$ are vectors of the same length, so we can compute their dot product using a fully connected layer.\n",
"\n",
"This model could have a performance issue as the softmax output covers millions of classification labels. To optimize performance, at the training time, the authors down-sample negative samples, so the actual number of classes is reduced to thousands. At serving time, the authors ignore the normalization of the softmax outputs, because the results are just for ranking.\n",
"\n",
"\n",
"#### Ranking Network\n",
"\n",
"The architecture of the ranking network is similar to that of the candidate generation network. Similar to ranking models widely used in online advertising, it uses rich features like video ID, last watching time, etc. The output layer of the ranking network is a weighted logistic regression, which rates all candidate videos.\n",
"\n",
"\n",
"### Hybrid Model\n",
"\n",
"In the section, let us introduce our movie recommendation system.\n",
"\n",
"In our network, the input includes features of users and movies. The user feature includes four properties: user ID, gender, occupation, and age. Movie features include their IDs, genres, and titles.\n",
"\n",
"We use fully-connected layers to map user features into representative feature vectors and concatenate them. The process of movie features is similar, except that for movie titles -- we feed titles into a text convolution network as described in the [sentiment analysis tutorial](https://github.com/PaddlePaddle/book/blob/develop/understand_sentiment/README.md)）to get a fixed-length representative feature vector.\n",
"\n",
"Given the feature vectors of users and movies, we compute the relevance using cosine similarity. We minimize the squared error at training time.\n",
"\n",
"\u003cp align=\"center\"\u003e\n",
"\n",
"\u003cimg src=\"image/rec_regression_network_en.png\" width=\"90%\" \u003e\u003cbr/\u003e\n",
"Figure 3. A hybrid recommendation model.\n",
"\u003c/p\u003e\n",
"\n",
"## Dataset\n",
"\n",
"We use the [MovieLens ml-1m](http://files.grouplens.org/datasets/movielens/ml-1m.zip) to train our model. This dataset includes 10,000 ratings of 4,000 movies from 6,000 users to 4,000 movies. Each rate is in the range of 1~5. Thanks to GroupLens Research for collecting, processing and publishing the dataset.\n",
"\n",
"`paddle.v2.datasets` package encapsulates multiple public datasets, including `cifar`, `imdb`, `mnist`, `moivelens` and `wmt14`, etc. There's no need for us to manually download and preprocess `MovieLens` dataset.\n",
"\n"
]
},
{
"cell_type": "code",
"metadata": {
"editable": true
},
"source": [
"# Run this block to show dataset's documentation\n",
"help(paddle.v2.dataset.movielens)\n"
],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n"
]
}
],
"execution_count": 1
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"The raw `MoiveLens` contains movie ratings, relevant features from both movies and users.\n",
"For instance, one movie's feature could be:\n",
"\n"
]
},
{
"cell_type": "code",
"metadata": {
"editable": true
},
"source": [
"movie_info = paddle.dataset.movielens.movie_info()\n",
"print movie_info.values()[0]\n"
],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n"
]
}
],
"execution_count": 1
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"```text\n",
"\u003cMovieInfo id(1), title(Toy Story), categories(['Animation', \"Children's\", 'Comedy'])\u003e\n",
"```\n",
"\n",
"One user's feature could be:\n",
"\n"
]
},
{
"cell_type": "code",
"metadata": {
"editable": true
},
"source": [
"user_info = paddle.dataset.movielens.user_info()\n",
"print user_info.values()[0]\n"
],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n"
]
}
],
"execution_count": 1
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"```text\n",
"\u003cUserInfo id(1), gender(F), age(1), job(10)\u003e\n",
"```\n",
"\n",
"In this dateset, the distribution of age is shown as follows:\n",
"\n",
"```text\n",
"1: \"Under 18\"\n",
"18: \"18-24\"\n",
"25: \"25-34\"\n",
"35: \"35-44\"\n",
"45: \"45-49\"\n",
"50: \"50-55\"\n",
"56: \"56+\"\n",
"```\n",
"\n",
"User's occupation is selected from the following options:\n",
"\n",
"```text\n",
"0: \"other\" or not specified\n",
"1: \"academic/educator\"\n",
"2: \"artist\"\n",
"3: \"clerical/admin\"\n",
"4: \"college/grad student\"\n",
"5: \"customer service\"\n",
"6: \"doctor/health care\"\n",
"7: \"executive/managerial\"\n",
"8: \"farmer\"\n",
"9: \"homemaker\"\n",
"10: \"K-12 student\"\n",
"11: \"lawyer\"\n",
"12: \"programmer\"\n",
"13: \"retired\"\n",
"14: \"sales/marketing\"\n",
"15: \"scientist\"\n",
"16: \"self-employed\"\n",
"17: \"technician/engineer\"\n",
"18: \"tradesman/craftsman\"\n",
"19: \"unemployed\"\n",
"20: \"writer\"\n",
"```\n",
"\n",
"Each record consists of three main components: user features, movie features and movie ratings.\n",
"Likewise, as a simple example, consider the following:\n",
"\n"
]
},
{
"cell_type": "code",
"metadata": {
"editable": true
},
"source": [
"train_set_creator = paddle.dataset.movielens.train()\n",
"train_sample = next(train_set_creator())\n",
"uid = train_sample[0]\n",
"mov_id = train_sample[len(user_info[uid].value())]\n",
"print \"User %s rates Movie %s with Score %s\"%(user_info[uid], movie_info[mov_id], train_sample[-1])\n"
],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n"
]
}
],
"execution_count": 1
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"```text\n",
"User \u003cUserInfo id(1), gender(F), age(1), job(10)\u003e rates Movie \u003cMovieInfo id(1193), title(One Flew Over the Cuckoo's Nest), categories(['Drama'])\u003e with Score [5.0]\n",
"```\n",
"\n",
"The output shows that user 1 gave movie `1193` a rating of 5.\n",
"\n",
"After issuing a command `python train.py`, training will start immediately. The details will be unpacked by the following sessions to see how it works.\n",
"\n",
"## Model Architecture\n",
"\n",
"### Initialize PaddlePaddle\n",
"\n",
"First, we must import and initialize PaddlePaddle (enable/disable GPU, set the number of trainers, etc).\n",
"\n"
]
},
{
"cell_type": "code",
"metadata": {
"editable": true
},
"source": [
"%matplotlib inline\n",
"\n",
"import matplotlib.pyplot as plt\n",
"from IPython import display\n",
"import cPickle\n",
"\n",
"import paddle.v2 as paddle\n",
"\n",
"paddle.init(use_gpu=False)\n"
],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n"
]
}
],
"execution_count": 1
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### Model Configuration\n",
"\n"
]
},
{
"cell_type": "code",
"metadata": {
"editable": true
},
"source": [
"uid = paddle.layer.data(\n",
" name='user_id',\n",
" type=paddle.data_type.integer_value(\n",
" paddle.dataset.movielens.max_user_id() + 1))\n",
"usr_emb = paddle.layer.embedding(input=uid, size=32)\n",
"\n",
"usr_gender_id = paddle.layer.data(\n",
" name='gender_id', type=paddle.data_type.integer_value(2))\n",
"usr_gender_emb = paddle.layer.embedding(input=usr_gender_id, size=16)\n",
"\n",
"usr_age_id = paddle.layer.data(\n",
" name='age_id',\n",
" type=paddle.data_type.integer_value(\n",
" len(paddle.dataset.movielens.age_table)))\n",
"usr_age_emb = paddle.layer.embedding(input=usr_age_id, size=16)\n",
"\n",
"usr_job_id = paddle.layer.data(\n",
" name='job_id',\n",
" type=paddle.data_type.integer_value(paddle.dataset.movielens.max_job_id(\n",
" ) + 1))\n",
"usr_job_emb = paddle.layer.embedding(input=usr_job_id, size=16)\n"
],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n"
]
}
],
"execution_count": 1
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"As shown in the above code, the input is four dimension integers for each user, that is, `user_id`,`gender_id`, `age_id` and `job_id`. In order to deal with these features conveniently, we use the language model in NLP to transform these discrete values into embedding vaules `usr_emb`, `usr_gender_emb`, `usr_age_emb` and `usr_job_emb`.\n",
"\n"
]
},
{
"cell_type": "code",
"metadata": {
"editable": true
},
"source": [
"usr_combined_features = paddle.layer.fc(\n",
" input=[usr_emb, usr_gender_emb, usr_age_emb, usr_job_emb],\n",
" size=200,\n",
" act=paddle.activation.Tanh())\n"
],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n"
]
}
],
"execution_count": 1
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"Then, employing user features as input, directly connecting to a fully-connected layer, which is used to reduce dimension to 200.\n",
"\n",
"Furthermore, we do a similar transformation for each movie feature. The model configuration is:\n",
"\n"
]
},
{
"cell_type": "code",
"metadata": {
"editable": true
},
"source": [
"mov_id = paddle.layer.data(\n",
" name='movie_id',\n",
" type=paddle.data_type.integer_value(\n",
" paddle.dataset.movielens.max_movie_id() + 1))\n",
"mov_emb = paddle.layer.embedding(input=mov_id, size=32)\n",
"\n",
"mov_categories = paddle.layer.data(\n",
" name='category_id',\n",
" type=paddle.data_type.sparse_binary_vector(\n",
" len(paddle.dataset.movielens.movie_categories())))\n",
"\n",
"mov_categories_hidden = paddle.layer.fc(input=mov_categories, size=32)\n",
"\n",
"\n",
"movie_title_dict = paddle.dataset.movielens.get_movie_title_dict()\n",
"mov_title_id = paddle.layer.data(\n",
" name='movie_title',\n",
" type=paddle.data_type.integer_value_sequence(len(movie_title_dict)))\n",
"mov_title_emb = paddle.layer.embedding(input=mov_title_id, size=32)\n",
"mov_title_conv = paddle.networks.sequence_conv_pool(\n",
" input=mov_title_emb, hidden_size=32, context_len=3)\n",
"\n",
"mov_combined_features = paddle.layer.fc(\n",
" input=[mov_emb, mov_categories_hidden, mov_title_conv],\n",
" size=200,\n",
" act=paddle.activation.Tanh())\n"
],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n"
]
}
],
"execution_count": 1
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"Movie title, a sequence of words represented by an integer word index sequence, will be feed into a `sequence_conv_pool` layer, which will apply convolution and pooling on time dimension. Because pooling is done on time dimension, the output will be a fixed-length vector regardless the length of the input sequence.\n",
"\n",
"Finally, we can use cosine similarity to calculate the similarity between user characteristics and movie features.\n",
"\n"
]
},
{
"cell_type": "code",
"metadata": {
"editable": true
},
"source": [
"inference = paddle.layer.cos_sim(a=usr_combined_features, b=mov_combined_features, size=1, scale=5)\n",
"cost = paddle.layer.regression_cost(\n",
" input=inference,\n",
" label=paddle.layer.data(\n",
" name='score', type=paddle.data_type.dense_vector(1)))\n"
],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n"
]
}
],
"execution_count": 1
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Model Training\n",
"\n",
"### Define Parameters\n",
"\n",
"First, we define the model parameters according to the previous model configuration `cost`.\n",
"\n"
]
},
{
"cell_type": "code",
"metadata": {
"editable": true
},
"source": [
"# Create parameters\n",
"parameters = paddle.parameters.create(cost)\n"
],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n"
]
}
],
"execution_count": 1
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### Create Trainer\n",
"\n",
"Before jumping into creating a training module, algorithm setting is also necessary. Here we specified Adam optimization algorithm via `paddle.optimizer`.\n",
"\n"
]
},
{
"cell_type": "code",
"metadata": {
"editable": true
},
"source": [
"trainer = paddle.trainer.SGD(cost=cost, parameters=parameters,\n",
" update_equation=paddle.optimizer.Adam(learning_rate=1e-4))\n"
],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n"
]
}
],
"execution_count": 1
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"```text\n",
"[INFO 2017-03-06 17:12:13,378 networks.py:1472] The input order is [user_id, gender_id, age_id, job_id, movie_id, category_id, movie_title, score]\n",
"[INFO 2017-03-06 17:12:13,379 networks.py:1478] The output order is [__regression_cost_0__]\n",
"```\n",
"\n",
"### Training\n",
"\n",
"`paddle.dataset.movielens.train` will yield records during each pass, after shuffling, a batch input is generated for training.\n",
"\n"
]
},
{
"cell_type": "code",
"metadata": {
"editable": true
},
"source": [
"reader=paddle.reader.batch(\n",
" paddle.reader.shuffle(\n",
" paddle.dataset.movielens.trai(), buf_size=8192),\n",
" batch_size=256)\n"
],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n"
]
}
],
"execution_count": 1
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"`feeding` is devoted to specifying the correspondence between each yield record and `paddle.layer.data`. For instance, the first column of data generated by `movielens.train` corresponds to `user_id` feature.\n",
"\n"
]
},
{
"cell_type": "code",
"metadata": {
"editable": true
},
"source": [
"feeding = {\n",
" 'user_id': 0,\n",
" 'gender_id': 1,\n",
" 'age_id': 2,\n",
" 'job_id': 3,\n",
" 'movie_id': 4,\n",
" 'category_id': 5,\n",
" 'movie_title': 6,\n",
" 'score': 7\n",
"}\n"
],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n"
]
}
],
"execution_count": 1
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"Callback function `event_handler` will be called during training when a pre-defined event happens.\n",
"\n"
]
},
{
"cell_type": "code",
"metadata": {
"editable": true
},
"source": [
"step=0\n",
"\n",
"train_costs=[],[]\n",
"test_costs=[],[]\n",
"\n",
"def event_handler(event):\n",
" global step\n",
" global train_costs\n",
" global test_costs\n",
" if isinstance(event, paddle.event.EndIteration):\n",
" need_plot = False\n",
" if step % 10 == 0: # every 10 batches, record a train cost\n",
" train_costs[0].append(step)\n",
" train_costs[1].append(event.cost)\n",
"\n",
" if step % 1000 == 0: # every 1000 batches, record a test cost\n",
" result = trainer.test(reader=paddle.batch(\n",
" paddle.dataset.movielens.test(), batch_size=256))\n",
" test_costs[0].append(step)\n",
" test_costs[1].append(result.cost)\n",
"\n",
" if step % 100 == 0: # every 100 batches, update cost plot\n",
" plt.plot(*train_costs)\n",
" plt.plot(*test_costs)\n",
" plt.legend(['Train Cost', 'Test Cost'], loc='upper left')\n",
" display.clear_output(wait=True)\n",
" display.display(plt.gcf())\n",
" plt.gcf().clear()\n",
" step += 1\n"
],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n"
]
}
],
"execution_count": 1
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"Finally, we can invoke `trainer.train` to start training:\n",
"\n"
]
},
{
"cell_type": "code",
"metadata": {
"editable": true
},
"source": [
"trainer.train(\n",
" reader=reader,\n",
" event_handler=event_handler,\n",
" feeding=feeding,\n",
" num_passes=200)\n"
],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n"
]
}
],
"execution_count": 1
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Conclusion\n",
"\n",
"This tutorial goes over traditional approaches in recommender system and a deep learning based approach. We also show that how to train and use the model with PaddlePaddle. Deep learning has been well used in computer vision and NLP, we look forward to its new successes in recommender systems.\n",
"\n",
"## Reference\n",
"\n",
"1. [Peter Brusilovsky](https://en.wikipedia.org/wiki/Peter_Brusilovsky) (2007). *The Adaptive Web*. p. 325.\n",
"2. Robin Burke , [Hybrid Web Recommender Systems](http://www.dcs.warwick.ac.uk/~acristea/courses/CS411/2010/Book%20-%20The%20Adaptive%20Web/HybridWebRecommenderSystems.pdf), pp. 377-408, The Adaptive Web, Peter Brusilovsky, Alfred Kobsa, Wolfgang Nejdl (Ed.), Lecture Notes in Computer Science, Springer-Verlag, Berlin, Germany, Lecture Notes in Computer Science, Vol. 4321, May 2007, 978-3-540-72078-2.\n",
"3. P. Resnick, N. Iacovou, etc. “[GroupLens: An Open Architecture for Collaborative Filtering of Netnews](http://ccs.mit.edu/papers/CCSWP165.html)”, Proceedings of ACM Conference on Computer Supported Cooperative Work, CSCW 1994. pp.175-186.\n",
"4. Sarwar, Badrul, et al. \"[Item-based collaborative filtering recommendation algorithms.](http://files.grouplens.org/papers/www10_sarwar.pdf)\" *Proceedings of the 10th International Conference on World Wide Web*. ACM, 2001.\n",
"5. Kautz, Henry, Bart Selman, and Mehul Shah. \"[Referral Web: Combining Social networks and collaborative filtering.](http://www.cs.cornell.edu/selman/papers/pdf/97.cacm.refweb.pdf)\" Communications of the ACM 40.3 (1997): 63-65. APA\n",
"6. Yuan, Jianbo, et al. [\"Solving Cold-Start Problem in Large-scale Recommendation Engines: A Deep Learning Approach.\"](https://arxiv.org/pdf/1611.05480v1.pdf) *arXiv preprint arXiv:1611.05480* (2016).\n",
"7. Covington P, Adams J, Sargin E. [Deep neural networks for youtube recommendations](https://static.googleusercontent.com/media/research.google.com/zh-CN//pubs/archive/45530.pdf)[C]//Proceedings of the 10th ACM Conference on Recommender Systems. ACM, 2016: 191-198.\n",
"\n",
"\u003cbr/\u003e\n",
"This tutorial is contributed by \u003ca xmlns:cc=\"http://creativecommons.org/ns#\" href=\"http://book.paddlepaddle.org\" property=\"cc:attributionName\" rel=\"cc:attributionURL\"\u003ePaddlePaddle\u003c/a\u003e, and licensed under a \u003ca rel=\"license\" href=\"http://creativecommons.org/licenses/by-nc-sa/4.0/\"\u003eCreative Commons Attribution-NonCommercial-ShareAlike 4.0 International License\u003c/a\u003e.\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.0"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
recommender_system/README.ipynb
浏览文件 @ 2f6da124
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"collapsed": true,
"deletable": true,
"editable": true
},
"source": [
"# 个性化推荐\n",
"\n",
"本教程源代码目录在[book/recommender_system](https://github.com/PaddlePaddle/book/tree/develop/recommender_system)， 初次使用请参考PaddlePaddle[安装教程](http://www.paddlepaddle.org/doc_cn/build_and_install/index.html)。\n",
"\n",
"## 背景介绍\n",
"\n",
"在网络技术不断发展和电子商务规模不断扩大的背景下，商品数量和种类快速增长，用户需要花费大量时间才能找到自己想买的商品，这就是信息超载问题。为了解决这个难题，推荐系统（Recommender System）应运而生。\n",
"\n",
"个性化推荐系统是信息过滤系统（Information Filtering System）的子集，它可以用在很多领域，如电影、音乐、电商和 Feed 流推荐等。推荐系统通过分析、挖掘用户行为，发现用户的个性化需求与兴趣特点，将用户可能感兴趣的信息或商品推荐给用户。与搜索引擎不同，推荐系统不需要用户准确地描述出自己的需求，而是根据分析历史行为建模，主动提供满足用户兴趣和需求的信息。\n",
"\n",
"传统的推荐系统方法主要有：\n",
"\n",
"- 协同过滤推荐（Collaborative Filtering Recommendation）：该方法收集分析用户历史行为、活动、偏好，计算一个用户与其他用户的相似度，利用目标用户的相似用户对商品评价的加权评价值，来预测目标用户对特定商品的喜好程度。优点是可以给用户推荐未浏览过的新产品；缺点是对于没有任何行为的新用户存在冷启动的问题，同时也存在用户与商品之间的交互数据不够多造成的稀疏问题，会导致模型难以找到相近用户。\n",
"- 基于内容过滤推荐[[1](#参考文献)]（Content-based Filtering Recommendation）：该方法利用商品的内容描述，抽象出有意义的特征，通过计算用户的兴趣和商品描述之间的相似度，来给用户做推荐。优点是简单直接，不需要依据其他用户对商品的评价，而是通过商品属性进行商品相似度度量，从而推荐给用户所感兴趣商品的相似商品；缺点是对于没有任何行为的新用户同样存在冷启动的问题。\n",
"- 组合推荐[[2](#参考文献)]（Hybrid Recommendation）：运用不同的输入和技术共同进行推荐，以弥补各自推荐技术的缺点。\n",
"\n",
"其中协同过滤是应用最广泛的技术之一，它又可以分为多个子类：基于用户 （User-Based）的推荐[[3](#参考文献)] 、基于物品（Item-Based）的推荐[[4](#参考文献)]、基于社交网络关系（Social-Based）的推荐[[5](#参考文献)]、基于模型（Model-based）的推荐等。1994年明尼苏达大学推出的GroupLens系统[[3](#参考文献)]一般被认为是推荐系统成为一个相对独立的研究方向的标志。该系统首次提出了基于协同过滤来完成推荐任务的思想，此后，基于该模型的协同过滤推荐引领了推荐系统十几年的发展方向。\n",
"\n",
"深度学习具有优秀的自动提取特征的能力，能够学习多层次的抽象特征表示，并对异质或跨域的内容信息进行学习，可以一定程度上处理推荐系统冷启动问题[[6](#参考文献)]。本教程主要介绍个性化推荐的深度学习模型，以及如何使用PaddlePaddle实现模型。\n",
"\n",
"## 效果展示\n",
"\n",
"我们使用包含用户信息、电影信息与电影评分的数据集作为个性化推荐的应用场景。当我们训练好模型后，只需要输入对应的用户ID和电影ID，就可以得出一个匹配的分数（范围[1,5]，分数越高视为兴趣越大），然后根据所有电影的推荐得分排序，推荐给用户可能感兴趣的电影。\n",
"\n",
"```\n",
"Input movie_id: 1962\n",
"Input user_id: 1\n",
"Prediction Score is 4.25\n",
"```\n",
"\n",
"## 模型概览\n",
"\n",
"本章中，我们首先介绍YouTube的视频推荐系统[[7](#参考文献)]，然后介绍我们实现的融合推荐模型。\n",
"\n",
"### YouTube的深度神经网络推荐系统\n",
"\n",
"YouTube是世界上最大的视频上传、分享和发现网站，YouTube推荐系统为超过10亿用户从不断增长的视频库中推荐个性化的内容。整个系统由两个神经网络组成：候选生成网络和排序网络。候选生成网络从百万量级的视频库中生成上百个候选，排序网络对候选进行打分排序，输出排名最高的数十个结果。系统结构如图1所示：\n",
"\n",
"<p align=\"center\">\n",
"<img src=\"image/YouTube_Overview.png\" width=\"70%\" ><br/>\n",
"图1. YouTube 推荐系统结构\n",
"</p>\n",
"\n",
"#### 候选生成网络（Candidate Generation Network）\n",
"\n",
"候选生成网络将推荐问题建模为一个类别数极大的多类分类问题：对于一个Youtube用户，使用其观看历史（视频ID）、搜索词记录（search tokens）、人口学信息（如地理位置、用户登录设备）、二值特征（如性别，是否登录）和连续特征（如用户年龄）等，对视频库中所有视频进行多分类，得到每一类别的分类结果（即每一个视频的推荐概率），最终输出概率较高的几百个视频。\n",
"\n",
"首先，将观看历史及搜索词记录这类历史信息，映射为向量后取平均值得到定长表示；同时，输入人口学特征以优化新用户的推荐效果，并将二值特征和连续特征归一化处理到[0, 1]范围。接下来，将所有特征表示拼接为一个向量，并输入给非线形多层感知器（MLP，详见[识别数字](https://github.com/PaddlePaddle/book/blob/develop/recognize_digits/README.md)教程）处理。最后，训练时将MLP的输出给softmax做分类，预测时计算用户的综合特征（MLP的输出）与所有视频的相似度，取得分最高的$k$个作为候选生成网络的筛选结果。图2显示了候选生成网络结构。\n",
"\n",
"<p align=\"center\">\n",
"<img src=\"image/Deep_candidate_generation_model_architecture.png\" width=\"70%\" ><br/>\n",
"图2. 候选生成网络结构\n",
"</p>\n",
"\n",
"对于一个用户$U$，预测此刻用户要观看的视频$\\omega$为视频$i$的概率公式为：\n",
"\n",
"$$P(\\omega=i|u)=\\frac{e^{v_{i}u}}{\\sum_{j \\in V}e^{v_{j}u}}$$\n",
"\n",
"其中$u$为用户$U$的特征表示，$V$为视频库集合，$v_i$为视频库中第$i$个视频的特征表示。$u$和$v_i$为长度相等的向量，两者点积可以通过全连接层实现。\n",
"\n",
"考虑到softmax分类的类别数非常多，为了保证一定的计算效率：1）训练阶段，使用负样本类别采样将实际计算的类别数缩小至数千；2）推荐（预测）阶段，忽略softmax的归一化计算（不影响结果），将类别打分问题简化为点积（dot product）空间中的最近邻（nearest neighbor）搜索问题，取与$u$最近的$k$个视频作为生成的候选。\n",
"\n",
"#### 排序网络（Ranking Network）\n",
"排序网络的结构类似于候选生成网络，但是它的目标是对候选进行更细致的打分排序。和传统广告排序中的特征抽取方法类似，这里也构造了大量的用于视频排序的相关特征（如视频 ID、上次观看时间等）。这些特征的处理方式和候选生成网络类似，不同之处是排序网络的顶部是一个加权逻辑回归（weighted logistic regression），它对所有候选视频进行打分，从高到底排序后将分数较高的一些视频返回给用户。\n",
"\n",
"### 融合推荐模型\n",
"\n",
"在下文的电影推荐系统中：\n",
"\n",
"1. 首先，使用用户特征和电影特征作为神经网络的输入，其中：\n",
"\n",
" - 用户特征融合了四个属性信息，分别是用户ID、性别、职业和年龄。\n",
"\n",
" - 电影特征融合了三个属性信息，分别是电影ID、电影类型ID和电影名称。\n",
"\n",
"2. 对用户特征，将用户ID映射为维度大小为256的向量表示，输入全连接层，并对其他三个属性也做类似的处理。然后将四个属性的特征表示分别全连接并相加。\n",
"\n",
"3. 对电影特征，将电影ID以类似用户ID的方式进行处理，电影类型ID以向量的形式直接输入全连接层，电影名称用文本卷积神经网络（详见[第5章](https://github.com/PaddlePaddle/book/blob/develop/understand_sentiment/README.md)）得到其定长向量表示。然后将三个属性的特征表示分别全连接并相加。\n",
"\n",
"4. 得到用户和电影的向量表示后，计算二者的余弦相似度作为推荐系统的打分。最后，用该相似度打分和用户真实打分的差异的平方作为该回归模型的损失函数。\n",
"\n",
"<p align=\"center\">\n",
"\n",
"<img src=\"image/rec_regression_network.png\" width=\"90%\" ><br/>\n",
"图3. 融合推荐模型 \n",
"</p> "
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": true,
"editable": true
},
"source": [
"## 数据准备\n",
"\n",
"### 数据介绍与下载\n",
"\n",
"我们以 [MovieLens 百万数据集（ml-1m）](http://files.grouplens.org/datasets/movielens/ml-1m.zip)为例进行介绍。ml-1m 数据集包含了 6,000 位用户对 4,000 部电影的 1,000,000 条评价（评分范围 1~5 分，均为整数），由 GroupLens Research 实验室搜集整理。\n",
"\n",
"Paddle在API中提供了自动加载数据的模块。数据模块为 `paddle.dataset.movielens`"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [],
"source": [
"import paddle.v2 as paddle\n",
"paddle.init(use_gpu=False)"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [],
"source": [
"# Run this block to show dataset's documentation\n",
"# help(paddle.dataset.movielens)"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": true,
"editable": true
},
"source": [
"在原始数据中包含电影的特征数据，用户的特征数据，和用户对电影的评分。\n",
"\n",
"例如，其中某一个电影特征为:"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [
"cells": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"<MovieInfo id(1), title(Toy Story ), categories(['Animation', \"Children's\", 'Comedy'])>\n"
]
}
],
"source": [
"movie_info = paddle.dataset.movielens.movie_info()\n",
"print movie_info.values()[0]"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": true,
"editable": true
},
"source": [
"这表示，电影的id是1，标题是《Toy Story》，该电影被分为到三个类别中。这三个类别是动画，儿童，喜剧。"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [
"cell_type": "markdown",
"metadata": {},
"source": [
"# 个性化推荐\n",
"\n",
"本教程源代码目录在[book/recommender_system](https://github.com/PaddlePaddle/book/tree/develop/recommender_system)， 初次使用请参考PaddlePaddle[安装教程](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/getstarted/build_and_install/docker_install_cn.rst)。\n",
"\n",
"## 背景介绍\n",
"\n",
"在网络技术不断发展和电子商务规模不断扩大的背景下，商品数量和种类快速增长，用户需要花费大量时间才能找到自己想买的商品，这就是信息超载问题。为了解决这个难题，推荐系统（Recommender System）应运而生。\n",
"\n",
"个性化推荐系统是信息过滤系统（Information Filtering System）的子集，它可以用在很多领域，如电影、音乐、电商和 Feed 流推荐等。推荐系统通过分析、挖掘用户行为，发现用户的个性化需求与兴趣特点，将用户可能感兴趣的信息或商品推荐给用户。与搜索引擎不同，推荐系统不需要用户准确地描述出自己的需求，而是根据分析历史行为建模，主动提供满足用户兴趣和需求的信息。\n",
"\n",
"传统的推荐系统方法主要有：\n",
"\n",
"- 协同过滤推荐（Collaborative Filtering Recommendation）：该方法收集分析用户历史行为、活动、偏好，计算一个用户与其他用户的相似度，利用目标用户的相似用户对商品评价的加权评价值，来预测目标用户对特定商品的喜好程度。优点是可以给用户推荐未浏览过的新产品；缺点是对于没有任何行为的新用户存在冷启动的问题，同时也存在用户与商品之间的交互数据不够多造成的稀疏问题，会导致模型难以找到相近用户。\n",
"- 基于内容过滤推荐[[1](#参考文献)]（Content-based Filtering Recommendation）：该方法利用商品的内容描述，抽象出有意义的特征，通过计算用户的兴趣和商品描述之间的相似度，来给用户做推荐。优点是简单直接，不需要依据其他用户对商品的评价，而是通过商品属性进行商品相似度度量，从而推荐给用户所感兴趣商品的相似商品；缺点是对于没有任何行为的新用户同样存在冷启动的问题。\n",
"- 组合推荐[[2](#参考文献)]（Hybrid Recommendation）：运用不同的输入和技术共同进行推荐，以弥补各自推荐技术的缺点。\n",
"\n",
"其中协同过滤是应用最广泛的技术之一，它又可以分为多个子类：基于用户 （User-Based）的推荐[[3](#参考文献)] 、基于物品（Item-Based）的推荐[[4](#参考文献)]、基于社交网络关系（Social-Based）的推荐[[5](#参考文献)]、基于模型（Model-based）的推荐等。1994年明尼苏达大学推出的GroupLens系统[[3](#参考文献)]一般被认为是推荐系统成为一个相对独立的研究方向的标志。该系统首次提出了基于协同过滤来完成推荐任务的思想，此后，基于该模型的协同过滤推荐引领了推荐系统十几年的发展方向。\n",
"\n",
"深度学习具有优秀的自动提取特征的能力，能够学习多层次的抽象特征表示，并对异质或跨域的内容信息进行学习，可以一定程度上处理推荐系统冷启动问题[[6](#参考文献)]。本教程主要介绍个性化推荐的深度学习模型，以及如何使用PaddlePaddle实现模型。\n",
"\n",
"## 效果展示\n",
"\n",
"我们使用包含用户信息、电影信息与电影评分的数据集作为个性化推荐的应用场景。当我们训练好模型后，只需要输入对应的用户ID和电影ID，就可以得出一个匹配的分数（范围[1,5]，分数越高视为兴趣越大），然后根据所有电影的推荐得分排序，推荐给用户可能感兴趣的电影。\n",
"\n",
"```\n",
"Input movie_id: 1962\n",
"Input user_id: 1\n",
"Prediction Score is 4.25\n",
"```\n",
"\n",
"## 模型概览\n",
"\n",
"本章中，我们首先介绍YouTube的视频推荐系统[[7](#参考文献)]，然后介绍我们实现的融合推荐模型。\n",
"\n",
"### YouTube的深度神经网络推荐系统\n",
"\n",
"YouTube是世界上最大的视频上传、分享和发现网站，YouTube推荐系统为超过10亿用户从不断增长的视频库中推荐个性化的内容。整个系统由两个神经网络组成：候选生成网络和排序网络。候选生成网络从百万量级的视频库中生成上百个候选，排序网络对候选进行打分排序，输出排名最高的数十个结果。系统结构如图1所示：\n",
"\n",
"\u003cp align=\"center\"\u003e\n",
"\u003cimg src=\"image/YouTube_Overview.png\" width=\"70%\" \u003e\u003cbr/\u003e\n",
"图1. YouTube 推荐系统结构\n",
"\u003c/p\u003e\n",
"\n",
"#### 候选生成网络（Candidate Generation Network）\n",
"\n",
"候选生成网络将推荐问题建模为一个类别数极大的多类分类问题：对于一个Youtube用户，使用其观看历史（视频ID）、搜索词记录（search tokens）、人口学信息（如地理位置、用户登录设备）、二值特征（如性别，是否登录）和连续特征（如用户年龄）等，对视频库中所有视频进行多分类，得到每一类别的分类结果（即每一个视频的推荐概率），最终输出概率较高的几百个视频。\n",
"\n",
"首先，将观看历史及搜索词记录这类历史信息，映射为向量后取平均值得到定长表示；同时，输入人口学特征以优化新用户的推荐效果，并将二值特征和连续特征归一化处理到[0, 1]范围。接下来，将所有特征表示拼接为一个向量，并输入给非线形多层感知器（MLP，详见[识别数字](https://github.com/PaddlePaddle/book/blob/develop/recognize_digits/README.md)教程）处理。最后，训练时将MLP的输出给softmax做分类，预测时计算用户的综合特征（MLP的输出）与所有视频的相似度，取得分最高的$k$个作为候选生成网络的筛选结果。图2显示了候选生成网络结构。\n",
"\n",
"\u003cp align=\"center\"\u003e\n",
"\u003cimg src=\"image/Deep_candidate_generation_model_architecture.png\" width=\"70%\" \u003e\u003cbr/\u003e\n",
"图2. 候选生成网络结构\n",
"\u003c/p\u003e\n",
"\n",
"对于一个用户$U$，预测此刻用户要观看的视频$\\omega$为视频$i$的概率公式为：\n",
"\n",
"$$P(\\omega=i|u)=\\frac{e^{v_{i}u}}{\\sum_{j \\in V}e^{v_{j}u}}$$\n",
"\n",
"其中$u$为用户$U$的特征表示，$V$为视频库集合，$v_i$为视频库中第$i$个视频的特征表示。$u$和$v_i$为长度相等的向量，两者点积可以通过全连接层实现。\n",
"\n",
"考虑到softmax分类的类别数非常多，为了保证一定的计算效率：1）训练阶段，使用负样本类别采样将实际计算的类别数缩小至数千；2）推荐（预测）阶段，忽略softmax的归一化计算（不影响结果），将类别打分问题简化为点积（dot product）空间中的最近邻（nearest neighbor）搜索问题，取与$u$最近的$k$个视频作为生成的候选。\n",
"\n",
"#### 排序网络（Ranking Network）\n",
"排序网络的结构类似于候选生成网络，但是它的目标是对候选进行更细致的打分排序。和传统广告排序中的特征抽取方法类似，这里也构造了大量的用于视频排序的相关特征（如视频 ID、上次观看时间等）。这些特征的处理方式和候选生成网络类似，不同之处是排序网络的顶部是一个加权逻辑回归（weighted logistic regression），它对所有候选视频进行打分，从高到底排序后将分数较高的一些视频返回给用户。\n",
"\n",
"### 融合推荐模型\n",
"\n",
"在下文的电影推荐系统中：\n",
"\n",
"1. 首先，使用用户特征和电影特征作为神经网络的输入，其中：\n",
"\n",
" - 用户特征融合了四个属性信息，分别是用户ID、性别、职业和年龄。\n",
"\n",
" - 电影特征融合了三个属性信息，分别是电影ID、电影类型ID和电影名称。\n",
"\n",
"2. 对用户特征，将用户ID映射为维度大小为256的向量表示，输入全连接层，并对其他三个属性也做类似的处理。然后将四个属性的特征表示分别全连接并相加。\n",
"\n",
"3. 对电影特征，将电影ID以类似用户ID的方式进行处理，电影类型ID以向量的形式直接输入全连接层，电影名称用文本卷积神经网络（详见[第5章](https://github.com/PaddlePaddle/book/blob/develop/understand_sentiment/README.md)）得到其定长向量表示。然后将三个属性的特征表示分别全连接并相加。\n",
"\n",
"4. 得到用户和电影的向量表示后，计算二者的余弦相似度作为推荐系统的打分。最后，用该相似度打分和用户真实打分的差异的平方作为该回归模型的损失函数。\n",
"\n",
"\u003cp align=\"center\"\u003e\n",
"\n",
"\u003cimg src=\"image/rec_regression_network.png\" width=\"90%\" \u003e\u003cbr/\u003e\n",
"图3. 融合推荐模型\n",
"\u003c/p\u003e\n",
"\n",
"## 数据准备\n",
"\n",
"### 数据介绍与下载\n",
"\n",
"我们以 [MovieLens 百万数据集（ml-1m）](http://files.grouplens.org/datasets/movielens/ml-1m.zip)为例进行介绍。ml-1m 数据集包含了 6,000 位用户对 4,000 部电影的 1,000,000 条评价（评分范围 1~5 分，均为整数），由 GroupLens Research 实验室搜集整理。\n",
"\n",
"Paddle在API中提供了自动加载数据的模块。数据模块为 `paddle.dataset.movielens`\n",
"\n",
"\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"<UserInfo id(1), gender(F), age(1), job(10)>\n"
]
}
],
"source": [
"user_info = paddle.dataset.movielens.user_info()\n",
"print user_info.values()[0]"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": true,
"editable": true
},
"source": [
"这表示，该用户ID是1，女性，年龄比18岁还年轻。职业ID是10。\n",
"\n",
"\n",
"其中，年龄使用下列分布\n",
"* 1: \"Under 18\"\n",
"* 18: \"18-24\"\n",
"* 25: \"25-34\"\n",
"* 35: \"35-44\"\n",
"* 45: \"45-49\"\n",
"* 50: \"50-55\"\n",
"* 56: \"56+\"\n",
"\n",
"职业是从下面几种选项里面选则得出:\n",
"* 0: \"other\" or not specified\n",
"* 1: \"academic/educator\"\n",
"* 2: \"artist\"\n",
"* 3: \"clerical/admin\"\n",
"* 4: \"college/grad student\"\n",
"* 5: \"customer service\"\n",
"* 6: \"doctor/health care\"\n",
"* 7: \"executive/managerial\"\n",
"* 8: \"farmer\"\n",
"* 9: \"homemaker\"\n",
"* 10: \"K-12 student\"\n",
"* 11: \"lawyer\"\n",
"* 12: \"programmer\"\n",
"* 13: \"retired\"\n",
"* 14: \"sales/marketing\"\n",
"* 15: \"scientist\"\n",
"* 16: \"self-employed\"\n",
"* 17: \"technician/engineer\"\n",
"* 18: \"tradesman/craftsman\"\n",
"* 19: \"unemployed\"\n",
"* 20: \"writer\""
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": true,
"editable": true
},
"source": [
"而对于每一条训练/测试数据，均为 <用户特征> + <电影特征> + 评分。\n",
"\n",
"例如，我们获得第一条训练数据:"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [
"cell_type": "code",
"metadata": {
"editable": true
},
"source": [
"import paddle.v2 as paddle\n",
"paddle.init(use_gpu=False)\n"
],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n"
]
}
],
"execution_count": 1
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"User <UserInfo id(1), gender(F), age(1), job(10)> rates Movie <MovieInfo id(1193), title(One Flew Over the Cuckoo's Nest ), categories(['Drama'])> with Score [5.0]\n"
]
}
],
"source": [
"train_set_creator = paddle.dataset.movielens.train()\n",
"train_sample = next(train_set_creator())\n",
"uid = train_sample[0]\n",
"mov_id = train_sample[len(user_info[uid].value())]\n",
"print \"User %s rates Movie %s with Score %s\"%(user_info[uid], movie_info[mov_id], train_sample[-1])"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": true,
"editable": true
},
"source": [
"即用户1对电影1193的评价为5分。"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": true,
"editable": true
},
"source": [
"## 模型配置说明\n",
"\n",
"下面我们开始根据输入数据的形式配置模型。"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"collapsed": true,
"deletable": true,
"editable": true
},
"outputs": [],
"source": [
"uid = paddle.layer.data(\n",
" name='user_id',\n",
" type=paddle.data_type.integer_value(\n",
" paddle.dataset.movielens.max_user_id() + 1))\n",
"usr_emb = paddle.layer.embedding(input=uid, size=32)\n",
"\n",
"usr_gender_id = paddle.layer.data(\n",
" name='gender_id', type=paddle.data_type.integer_value(2))\n",
"usr_gender_emb = paddle.layer.embedding(input=usr_gender_id, size=16)\n",
"\n",
"usr_age_id = paddle.layer.data(\n",
" name='age_id',\n",
" type=paddle.data_type.integer_value(\n",
" len(paddle.dataset.movielens.age_table)))\n",
"usr_age_emb = paddle.layer.embedding(input=usr_age_id, size=16)\n",
"\n",
"usr_job_id = paddle.layer.data(\n",
" name='job_id',\n",
" type=paddle.data_type.integer_value(paddle.dataset.movielens.max_job_id(\n",
" ) + 1))\n",
"usr_job_emb = paddle.layer.embedding(input=usr_job_id, size=16)"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": true,
"editable": true
},
"source": [
"如上述代码所示，对于每个用户，我们输入4维特征。其中包括`user_id`,`gender_id`,`age_id`,`job_id`。这几维特征均是简单的整数值。为了后续神经网络处理这些特征方便，我们借鉴NLP中的语言模型，将这几维离散的整数值，变换成embedding取出。分别形成`usr_emb`, `usr_gender_emb`, `usr_age_emb`, `usr_job_emb`。"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"collapsed": true,
"deletable": true,
"editable": true
},
"outputs": [],
"source": [
"usr_combined_features = paddle.layer.fc(\n",
" input=[usr_emb, usr_gender_emb, usr_age_emb, usr_job_emb],\n",
" size=200,\n",
" act=paddle.activation.Tanh())"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": true,
"editable": true
},
"source": [
"然后，我们对于所有的用户特征，均输入到一个全连接层(fc)中。将所有特征融合为一个200维度的特征。"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": true,
"editable": true
},
"source": [
"进而，我们对每一个电影特征做类似的变换，网络配置为:"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [],
"source": [
"mov_id = paddle.layer.data(\n",
" name='movie_id',\n",
" type=paddle.data_type.integer_value(\n",
" paddle.dataset.movielens.max_movie_id() + 1))\n",
"mov_emb = paddle.layer.embedding(input=mov_id, size=32)\n",
"\n",
"mov_categories = paddle.layer.data(\n",
" name='category_id',\n",
" type=paddle.data_type.sparse_binary_vector(\n",
" len(paddle.dataset.movielens.movie_categories())))\n",
"\n",
"mov_categories_hidden = paddle.layer.fc(input=mov_categories, size=32)\n",
"\n",
"\n",
"movie_title_dict = paddle.dataset.movielens.get_movie_title_dict()\n",
"mov_title_id = paddle.layer.data(\n",
" name='movie_title',\n",
" type=paddle.data_type.integer_value_sequence(len(movie_title_dict)))\n",
"mov_title_emb = paddle.layer.embedding(input=mov_title_id, size=32)\n",
"mov_title_conv = paddle.networks.sequence_conv_pool(\n",
" input=mov_title_emb, hidden_size=32, context_len=3)\n",
"\n",
"mov_combined_features = paddle.layer.fc(\n",
" input=[mov_emb, mov_categories_hidden, mov_title_conv],\n",
" size=200,\n",
" act=paddle.activation.Tanh())"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": true,
"editable": true
},
"source": [
"电影ID和电影类型分别映射到其对应的特征隐层。对于电影标题名称(title)，一个ID序列表示的词语序列，在输入卷积层后，将得到每个时间窗口的特征（序列特征），然后通过在时间维度降采样得到固定维度的特征，整个过程在text_conv_pool实现。\n",
"\n",
"最后再将电影的特征融合进`mov_combined_features`中。"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"collapsed": true,
"deletable": true,
"editable": true
},
"outputs": [],
"source": [
"inference = paddle.layer.cos_sim(a=usr_combined_features, b=mov_combined_features, size=1, scale=5)"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": true,
"editable": true
},
"source": [
"进而，我们使用余弦相似度计算用户特征与电影特征的相似性。并将这个相似性拟合(回归)到用户评分上。"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"collapsed": true,
"deletable": true,
"editable": true
},
"outputs": [],
"source": [
"cost = paddle.layer.regression_cost(\n",
" input=inference,\n",
" label=paddle.layer.data(\n",
" name='score', type=paddle.data_type.dense_vector(1)))"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": true,
"editable": true
},
"source": [
"至此，我们的优化目标就是这个网络配置中的`cost`了。"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": true,
"editable": true
},
"source": [
"## 训练模型\n",
"\n",
"### 定义参数\n",
"神经网络的模型，我们可以简单的理解为网络拓朴结构+参数。之前一节，我们定义出了优化目标`cost`。这个`cost`即为网络模型的拓扑结构。我们开始训练模型，需要先定义出参数。定义方法为:"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"[INFO 2017-03-06 17:12:13,284 networks.py:1472] The input order is [user_id, gender_id, age_id, job_id, movie_id, category_id, movie_title, score]\n",
"[INFO 2017-03-06 17:12:13,287 networks.py:1478] The output order is [__regression_cost_0__]\n"
]
}
],
"source": [
"parameters = paddle.parameters.create(cost)"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": true,
"editable": true
},
"source": [
"`parameters`是模型的所有参数集合。他是一个python的dict。我们可以查看到这个网络中的所有参数名称。因为之前定义模型的时候，我们没有指定参数名称，这里参数名称是自动生成的。当然，我们也可以指定每一个参数名称，方便日后维护。"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [
"cell_type": "code",
"metadata": {
"editable": true
},
"source": [
"# Run this block to show dataset's documentation\n",
"# help(paddle.dataset.movielens)\n"
],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n"
]
}
],
"execution_count": 1
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"[u'___fc_layer_2__.wbias', u'___fc_layer_2__.w2', u'___embedding_layer_3__.w0', u'___embedding_layer_5__.w0', u'___embedding_layer_2__.w0', u'___embedding_layer_1__.w0', u'___fc_layer_1__.wbias', u'___fc_layer_0__.wbias', u'___fc_layer_1__.w0', u'___fc_layer_0__.w2', u'___fc_layer_0__.w3', u'___fc_layer_0__.w0', u'___fc_layer_0__.w1', u'___fc_layer_2__.w1', u'___fc_layer_2__.w0', u'___embedding_layer_4__.w0', u'___sequence_conv_pool_0___conv_fc.w0', u'___embedding_layer_0__.w0', u'___sequence_conv_pool_0___conv_fc.wbias']\n"
]
}
],
"source": [
"print parameters.keys()"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": true,
"editable": true
},
"source": [
"### 构造训练(trainer)\n",
"\n",
"下面，我们根据网络拓扑结构和模型参数来构造出一个本地训练(trainer)。在构造本地训练的时候，我们还需要指定这个训练的优化方法。这里我们使用Adam来作为优化算法。"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"在原始数据中包含电影的特征数据，用户的特征数据，和用户对电影的评分。\n",
"\n",
"例如，其中某一个电影特征为:\n",
"\n",
"\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"[INFO 2017-03-06 17:12:13,378 networks.py:1472] The input order is [user_id, gender_id, age_id, job_id, movie_id, category_id, movie_title, score]\n",
"[INFO 2017-03-06 17:12:13,379 networks.py:1478] The output order is [__regression_cost_0__]\n"
]
}
],
"source": [
"trainer = paddle.trainer.SGD(cost=cost, parameters=parameters, \n",
" update_equation=paddle.optimizer.Adam(learning_rate=1e-4))"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": true,
"editable": true
},
"source": [
"### 训练\n",
"\n",
"下面我们开始训练过程。\n",
"\n",
"我们直接使用Paddle提供的数据集读取程序。`paddle.dataset.movielens.train()`和`paddle.dataset.movielens.test()`分别做训练和预测数据集。并且通过`reader_dict`来指定每一个数据和data_layer的对应关系。\n",
"\n",
"例如，这里的reader_dict表示的是，对于数据层 `user_id`，使用了reader中每一条数据的第0个元素。`gender_id`数据层使用了第1个元素。以此类推。\n",
"\n",
"训练过程是完全自动的。我们可以使用event_handler来观察训练过程，或进行测试等。这里我们在event_handler里面绘制了训练误差曲线和测试误差曲线。并且保存了模型。"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [
"cell_type": "code",
"metadata": {
"editable": true
},
"source": [
"movie_info = paddle.dataset.movielens.movie_info()\n",
"print movie_info.values()[0]\n"
],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n"
]
}
],
"execution_count": 1
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD8CAYAAABn919SAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzsnXd4HNX1v987s0VarbptuVvucpcLYGOaKQm2CRACKQQI\nEEpCEgL8EkIKIUC+hEBCEkIIMSWVTmgBjI0pxmCwccW9N7mrd22b3x+zMzu7Oyutula+7/P4sXbq\n3dndzz333HPOFZqmIZFIJJLUR+nuBkgkEomkY5CCLpFIJL0EKegSiUTSS5CCLpFIJL0EKegSiUTS\nS5CCLpFIJL0EKegSiUTSS5CCLpFIJL0EKegSiUTSS3B05c369OmjFRYWduUtJRKJJOVZvXp1qaZp\nfVs6rksFvbCwkFWrVnXlLSUSiSTlEULsS+Y46XKRSCSSXoIUdIlEIuklSEGXSCSSXkKX+tDt8Pv9\nlJSU0NjY2N1N6TWkpaUxePBgnE5ndzdFIpF0Id0u6CUlJWRmZlJYWIgQorubk/JomkZZWRklJSUM\nHz68u5sjkUi6kG53uTQ2NpKfny/FvIMQQpCfny9HPBLJCUi3CzogxbyDkc9TIjkx6RGC3hIVdT7K\napu6uxkSiUTSo0kJQa9q8FNe5+uUa5eVlVFcXExxcTH9+/dn0KBB5mufL7l7XnPNNWzbtq1V933z\nzTeZPn06EyZMoLi4mJ/85CetbvuaNWt4++23W32eRCLpnXT7pGgyCAGhTlrLOj8/n3Xr1gHwq1/9\nCq/Xy49+9KOoYzRNQ9M0FMW+//v73//eqnuuX7+eW265hTfffJMxY8YQDAZZsGBBq9u+Zs0aNm7c\nyPnnn9/qcyUSSe8jJSx0RQhCWicpegJ27tzJ+PHj+eY3v8mECRM4fPgwN9xwAzNmzGDChAncc889\n5rGnnXYa69atIxAIkJOTwx133MGUKVOYNWsWx44di7v2b3/7W+68807GjBkDgKqqfPe73wVgz549\nzJkzh8mTJ3PeeedRUlICwHPPPcfEiROZMmUKc+bMoaGhgXvuuYenn36a4uJiXnrppS54KhKJpCfT\noyz0u/+3ic2HquO2+wIhAqEQHlfrmzt+YBZ3fWlCm9qzdetW/vWvfzFjxgwA7r//fvLy8ggEAsyZ\nM4dLL72U8ePHR51TVVXFmWeeyf33389tt93GU089xR133BF1zMaNG/n5z39ue8+bbrqJ6667jm9+\n85ssWLCAW265hZdeeom7776bDz74gIKCAiorK0lPT+eXv/wlGzdu5I9//GOb3p9EIuldpISFjoCu\ntc91Ro4caYo5wLPPPsu0adOYNm0aW7ZsYfPmzXHnpKenM3fuXACmT5/O3r17W3XPFStW8PWvfx2A\nq666imXLlgEwe/ZsrrrqKp544glCoVAb35FEIunN9CgLPZElfbS6kaPVjUwalN2lIXkZGRnm3zt2\n7OBPf/oTK1euJCcnhyuuuMI21tvlcpl/q6pKIBCIO2bChAmsXr2aCROSHzk8/vjjrFixgjfeeINp\n06axdu3aVr4biUTS20kJC10Ja3hnTYwmQ3V1NZmZmWRlZXH48GEWLVrU5mvdfvvt3HvvvezcuROA\nYDDIY489BsDMmTN54YUXAPjPf/7DGWecAcDu3buZOXMm9957L7m5uRw8eJDMzExqamra+c4kEklv\noUdZ6IkwrPKQpqHSPUkz06ZNY/z48RQVFTFs2DBmz57d5mtNnTqV3//+93z1q181rfyLLroIgL/8\n5S9ce+21/OY3v6GgoMCMoLn11lvZs2cPmqbxhS98gYkTJ1JQUMCDDz7I1KlT+fnPf86ll17a/jcq\nkUhSFqF1YfTIjBkztNgFLrZs2cK4ceOaPa+8zkdJRT1F/TNxOdTObGKvIZnnKpFIUgMhxGpN02a0\ndJx0uUgkEkkvIUUEPeJykUgkEok9qSHoYRM9JE10iUQiSUhKCLoattCDUtAlEokkIakh6GELPShd\nLhKJRJKQ1BJ0aaFLJBJJQlJC0BUBAtEpgt4R5XMBnnrqKY4cOWK7T9M0HnjgAcaOHUtxcTEnnXQS\nTz/9dKvb+vLLL7N169ZWnyeRSE4MUiaxSFU6R9CTKZ+bDE899RTTpk2jf//+cfv+8pe/8P7777Nq\n1SoyMzOpqqritddea/U9Xn75ZRRFoaioqNXnSiSS3k9KWOhAWNC79p7//Oc/OfnkkykuLuamm24i\nFAoRCAS48sormTRpEhMnTuThhx/m+eefZ926dXzta1+ztezvu+8+HnvsMTIzMwHIzs7mqquuAmDx\n4sUUFxczadIkrr/+evPcH//4x4wfP57Jkyfzk5/8hGXLlvHWW29x6623Ulxc3OqiXxKJpPfTsyz0\nhXfAkQ22u4b6A4AAZyszRftPgrn3t7opGzdu5JVXXmH58uU4HA5uuOEGnnvuOUaOHElpaSkbNujt\nrKysJCcnhz//+c888sgjFBcXR12nvLwcv9/PsGHD4u5RX1/Ptddey9KlSxk5cqRZMveyyy7jrbfe\nYtOmTQghzHvMmzePSy+9lIsvvrjV70cikfR+UsNC14IodK15vmTJEj777DNmzJhBcXExS5cuZdeu\nXYwaNYpt27Zx8803s2jRIrKzs9t8jy1btjBmzBhGjhwJ6OVyP/zwQ/Ly8lAUheuvv55XXnklquqj\nRCKRJKJnWeiJLOnSnSj+JvaKoYztn9klTdE0jWuvvZZ77703bt/nn3/OwoUL+ctf/sJ///vfZpeP\ny8vLw+l0sn//foYOHZrUvZ1OJ6tWreKdd97hxRdf5K9//SuLFy9u83uRSCQnBqlhoadl49R8qKGm\nLrvlueeeywsvvEBpaSmgR8Ps37+f48ePo2kal112Gffccw9r1qwBaLaU7R133MFNN91k7q+urubf\n//4348aNY8eOHezevRvQy+WeeeaZ1NTUUF1dzQUXXMAf/vAHs/a5LJcrkUiao2dZ6IlIy4bqErxa\nHZqW3yWLXEyaNIm77rqLc889l1AohNPp5LHHHkNVVb797W+jaRpCCH77298CcM0113DdddeRnp7O\nypUroxa6+MEPfkBdXR3Tp0/H5XLhdDq5/fbb8Xg8PPnkk1xyySUEg0FOOeUUrr/+eo4dO8Yll1xC\nU1MToVCIhx56CIBvfOMb3Hjjjfz+97/n1VdfpbCwsNOfg0QiSR1aLJ8rhHgKuAA4pmnaxPC2POB5\noBDYC3xV07SKlm7W1vK5AP4jW/AFNdIGjDMTjSSJkeVzJZLeQ0eWz/0HcH7MtjuAdzVNGw28G37d\nqfidmXhoIhRIPtlHIpFITiRaFHRN0z4EymM2XwT8M/z3P4FOj6MLuLIQAmiq7uxbSSQSSUrS1knR\nAk3TDof/PgIUtKcRSa2a5EjHpzkQjVXtudUJQVeuQiWRSHoO7Y5y0XT1SKggQogbhBCrhBCrjh8/\nHrc/LS2NsrKyFkVIUQTVeFD9tRAKtrfZvRZN0ygrKyMtLa27myKRSLqYtka5HBVCDNA07bAQYgBw\nLNGBmqYtABaAPikau3/w4MGUlJRgJ/ZWfIEQVTU19BVVUBoEp6eNTe/9pKWlMXjw4O5uhkQi6WLa\nKuivA98C7g//3/pKU2GcTifDhw9v8bh9ZXV85cElbM78Pq7xF8CX/9rWW0okEkmvpEWXixDiWeAT\nYKwQokQI8W10IT9PCLEDODf8ulPJcDsI4OBAn9Ng+9sQDHT2LSUSiSSlaNFC1zTtGwl2ndPBbWkW\nr1tv6o7cMxh5+C04sAIKZ3dlEyQSiaRHkxqp/4DboeBQBFszTgbVBdve6u4mSSQSSY8iZQRdCIHL\noVCrpcPwM2DrmyDD8yQSicQkZQQdwKkq+IMhGDsPKvbAsS3d3SSJRCLpMaScoPuCmi7oANve7N4G\nSSQSSQ8ipQTdpQrdQs8aAIOmw1bpR5dIJBKDlBJ0p0MhYCwsOnYeHFoD1YebP0kikUhOEFJL0FUF\nfzA8EVo0X/9fRrtIJBIJkIKC7jMs9L5FkDtcCrpEIpGESSlBN33oAELoVvqeD6FJLssmkUgkKSXo\nZtiiwdh5EPTBziXd1yiJRCLpIaSeoAcsyURDToH0PBntIpFIJKSaoDssPnQA1QFjzocdiyDo776G\nSSQSSQ8gpQQ9yoduUDQPGqtg3/LuaZREIpH0EFJK0ON86AAjzwZHmox2kUgkJzwpKOgxBblcGTDi\nLN2PLot1SSSSE5iUE3RfIBS/Y+w8qNoPRzZ0faMkEomkh5BSgu5yCAIhO0GfCwjpdpFIJCc0KSXo\nti4XAG8/GHKyXiNdIpFITlBST9DtXC6gu12OfA6VB7q2URKJRNJDSClBdzkUGgNB+51msa6FXdcg\niUQi6UGklKB73Q78QY0mO1HvMxryR8tFLyQSyQlLSgl6ZpoDgJrGgP0BRfNg70fQUNmFrZJIJJKe\nQe8S9LHzIRSQxbokEskJSWoJutsJQE1jgrotg2dARl8Z7SKRSE5IUkvQW7LQFVUv1rVzCQR8Xdgy\niUQi6X5STNBbsNBBj3Zpqoa9y7qoVRKJRNIzSDFB1y306kQWOuh1XZwemTUqkUhOOFJS0GubE3Rn\nul6BURbrkkgkJxgpJehedws+dIOx86DmEBxa2wWtkkgkkp5BSgm6Q1XwuNTmfeigT4wKRbpdJBLJ\nCUVKCTroVnqLFnpGPgydJdcalUgkJxQpJ+iZaQ5qmpJYP3TsPDi2CSr2dnqbJBKJpCeQgoLubNlC\nB70MAEgrXSKRnDCkoKAn4XIByBsBfcdJP7pEIjlhSElBr21KQtBBt9L3LYf68s5tlEQikfQAUk7Q\n3Q7VvnyuHWPngxaEHYs7t1ESiUTSA2iXoAshbhVCbBJCbBRCPCuESOuohiXClWihaDsGTgVvf1ms\nSyKRnBC0WdCFEIOAm4EZmqZNBFTg6x3VsES4HK0QdEXRF5De+S74Gzu3YRKJRNLNtNfl4gDShRAO\nwAMcan+TmqdVgg56sS5/Hez5sPMaJZFIJD2ANgu6pmkHgd8B+4HDQJWmaXHOaiHEDUKIVUKIVceP\nH297S8O4HAq+YCsEffgZ4PLKpekkEkmvpz0ul1zgImA4MBDIEEJcEXucpmkLNE2boWnajL59+7a9\npWHcDgV/UCMUSrLwlsMNo87RF48OtaIjkEgkkhSjPS6Xc4E9mqYd1zTND7wMnNoxzUqMy6E3uVVW\netEFUHsUDq7upFZJJBJJ99MeQd8PzBRCeIQQAjgH2NIxzUqMS9Wb3NQaP/ro80Co0u0ikUh6Ne3x\noa8AXgLWABvC11rQQe1KiNuw0AMhqhv9yble0nOhcLYsAyCRSHo17Ypy0TTtLk3TijRNm6hp2pWa\npjV1VMMS4XaoAFTU+5j8q8Xc//bW5E4cOx9Kt0HZrk5snUQikXQfKZcpavjQS2v1vuP1dUlGSprF\nuqTbRSKR9E5SVtD9Qd3VEkp2mbmcoVAwSRbrkkgkvZbUE/TwpKg/PCnaqlVDi+bBgRVQV9rxDZNI\nJJJuJuUE3e3Um7z5cDXQynWgx84DLQTb3+6ElkkkEkn3knKC7lD0Jj/0zvbwllYo+oApkDVYRrtI\nJJJeScoJ+uTB2QCkhS31VlnoQujFuna9B776TmidRCKRdB8pJ+gZbgcnD88jJ90FtNKHDrofPdAA\nuz/o6KZJJBJJt5Jygg7gVIW5yIXWKhMdGHYauLNk1qhEIul1pKSgOxTFTP1PtkZX5GQXjP4CbHsb\nQkmufCSRSCQpQEoKum6hh8MWW2uhg+52qS+FAys7uGUSiUTSfaSkoDsUhWDYNG+DnMOo80BxSreL\nRCLpVaSmoKsi8qItip6WBcNP18MX22LhSyQSSQ8kJQXdqUaa3WY5HjsPyndB6faWj5VIJJIUICUF\n3aFELPQ2+dBBF3SQxbokEkmvITUFvSMs9OxBMKBYFuuSSCS9hpQUdKfFh550tUU7iuZDySqoOdoB\nrZJIJJLuJSUF3ajnAu2c0xw7D9Bg+8J2t0kikUi6m5QUdKuF3q4YlYIJep10WaxLIpH0AlJS0Nsd\ntmgghL403e4PoKm2vc2SSCSSbiU1Bd3qcmmfja5njQab9AqMEolEksKkpKBHuVzamxc09FRIy5HR\nLhKJJOVJSUHvkLBFA9UBY87XVzEKBtp7NYlEIuk2UlPQOyKxyErRPGiogP2ftP9aEolE0k2kpKC7\nHJFmt7p8rh0jzwHVLd0uEokkpUlJQVeEaPmg1uD2wogz9TIAsliXRCJJUVJS0Hcfr+v4i46dB5X7\n4Njmjr+2RCKRdAEpKegnFeZ2/EXHztX/l0lGEokkRUlJQZ87aUDU60Aw1P6LZvaHQTPkohcSiSRl\nSUlBj6W6sYPCDYvmwaG1UH2oY64nkUgkXUivEPTKel/HXGjsfP1/Ge0ikUhSkF4h6C+sKumYC/Ud\nC3kjpB9dIpGkJL1C0B9buqtjLiSEHu2y50NorO6Ya0okEkkX0SsEvV+mu+MuVjQfQn7YuaTjrimR\nSCRdQMoL+swReQzISe+4Cw45BTz50o8ukUhSjpQVdMMqdzlUQh2S/x9GUWHMXNi+GIL+jruuRCKR\ndDLtEnQhRI4Q4iUhxFYhxBYhxKyOalhL/Pe7p/LApZNxqQqBjhR00MMXm6pg70cde12JRCLpRNpr\nof8JeFvTtCJgCrCl/U1KjiF5Hr46YwgORXSshQ4wYg440qXbRSKRpBRtFnQhRDZwBvAkgKZpPk3T\nKjuqYcmiqoJAqAMyRa24PDByjh6+KIt1SSSSFKE9Fvpw4DjwdyHEWiHEE0KIjA5qV9KoQhDsaAsd\n9PDF6hI48nnHX1sikUg6gfYIugOYBvxV07SpQB1wR+xBQogbhBCrhBCrjh8/3o7bJWiEIgh2hhU9\n5nxAyCQjiUSSMrRH0EuAEk3TVoRfv4Qu8FFomrZA07QZmqbN6Nu3bztuZ4+qCILBThB0b189hFEW\n65JIJClCmwVd07QjwAEhxNjwpnOALi8mriqi46NcDIrmwZENULm/c64vkUgkHUh7o1x+ADwthPgc\nKAbua3+TWoeqCEKdNXFpFuta2DnXl0gkkg6kXYKuadq6sDtlsqZpF2uaVtFRDUsWR2da6H1GQZ8x\n+tJ0EolE0sNJ2UxRA6WzfOgGY+fBvo+hocsjMiUSiaRVpLygd1qUi0HRfAgFYMc7nXcPiUQi6QBS\nXtBVpf2p/8dqGimvS7BIxqAZkNGPmvWvUXjHm6zZ3+VeJYlEIkmKXiDotDux6OT/e5dp9yawwBUF\nxs7Fvfc9XPh5Y/3hdt1LIpFIOoteIOgKwZCG1sluF1ewjlnK5o4vMyCRSCQdRMoLukMRAHRWoAsA\nw8/Er6ZznrIKf2dOwEokEkk7SHlBV8OC3qmWszONI31nc666hlAw0Hn3kUgkknbQawQ9FIKbn13L\n5Y9/anvc/rJ6bn52LfW+tgny4f5n019U0L9ua5vbKpFIJJ1Jygu6w2Khv77+EMt3lQHgD4ai/Opf\nW/AJr68/xJbDNQmvdeerG9ly2H5x6NKBZxHQFMZVy0UvJBJJzyTlBV0RuqDf91ZkbY3S2iZG/3wh\n/1y+19x2uKoRAF8gsWvm35/u4+q/r7Tdp6XlsUoby6Tajzug1RKJRNLxpLygO1Rd0J9decDctr+8\nHoBX1h2KO74ll8vR6ibqmuKPURXBO8HpDPLtgfI97WmypAto9Af5dHdZdzdDIulSUl7QDQvdSiAc\nieJQBOsOVHLP/zZjHFbvC7Z4zQcXbbPZqrE4NF3/Uy5N1+N5cNE2vr7gUzYfsnehSSS9kZQX9EAw\n3oXiD29TFcFX//YJT328x1xJriEJQa9u8MdtC2lwQCvggKNQLnqRAhyt1l1sieZEJJLeSMoLer0/\nXqAr63VBdqoCNcaCf339oSh/ux12keZGNupnaTNh/3KoL29bgyVdwpA8DwD7wu43ieREIOUF3c7i\nPl6jW2eqophRMAYf7SxlwYe7m72mXdapUXN9hWsWaCHYvqitTe6xNPqDbQ7r7Gk4Vf2rXW8zHyKR\n9FZSXtDtfOKltXqhLaciUNV4H3tbMAR9qzIKMgf2yqXp5vzuA8b/snd0VE3hkVsgpPHP5XubjW6y\ncqiygVfWlnRm0ySSTsPR3Q1oL3aCfrymCdAjYGIt9GSwd7no/wdCGoydC+ufA38DONNbff2eihHa\n2RtoDAv6858doMEfpLLezw/PHd3ieZc//il7y+qZO3EAaU61s5spkXQoKW+hn1SYG7ft+VV6CKND\nUcxM0lhaW8zLsND9wZC+1qi/DnYvbWVrJV1FU9gibwgLe3ldU1LnHQlPpra3gqdE0h2kvKB/eeog\nHv7GVNt9GhoOxf4ttraGeih8fFWDHwpPB1dmr3S7QO8Qs8aYyfJkP29BOPNYFmGTpCApL+hCCAZm\np5mvf3fZFPNvXyBkJh4BWI315n6wdsa7sSpSRb0fHG4YfS5se1svItPLsAvbtEPTOrlscTto9Ed/\nLq0VaH8v/FwlvZ+UF3QAlyPyNpwWAW8KhKJ86F+ZNtj822cTv27w+vpDvLwmemLMMPDMybWx86Hu\nGBxc1Z6m90gqkxT04T99i5+9srGTW9M2GgNttNDDXxdpoUtSkV4h6EaIGhDlYmkKhKL2zSjM5YLJ\nAwD7hCQrt72wXnevhAlZBKHeF4DR54HigK29w+1iJOJA8v5mgGdX7u+M5rSbplgLPUmL2+j+/S18\nPySSnkivEPTmLHS3ZZ/LoXDqyD5AxGJrzmVgjcm2+pXLan2QngPDZsO2tyipqOf8P37IzmOJKzn2\nVKrq/VTV+7nk0eXmth1Ha1s8r6tdLcdrmthQUpX08W230I3qndJCl6QevUPQVaugR/5ef6CS7RZx\nqm0MmD51w3XSnC5Zk5ZCmtVCD28vmg+l29m2aS1bj9Twk/9uaNf76A5O+c0S5vz+Aw5WNpjbNiVR\n/6Sr9W7un5bxpUeSL10c70NvncXd2uMlkp5ArxB0qxXuiEkkavAHOXdcAbedN4YLiweZFvzd/9tM\noz9oTnYa/OOak7j3oglAdIy7VdCbDOtv7FwAhh7/AIDV+yo65g21g0AwxCWPfsyH24/H7dtfVs8b\nn0cqUJZU1NPoD1Fe54s6zup+SURXuyRKa5N3A0F8+5KJ3Fm1N1LOIdWXGgyGNKobk5sLkfQeeoWg\nJ/KhG2SlO7j5nNFkpzvN/Uu2HOU/n+6L+6FPHZpLYZ8MIBLDDNEWqWn95QyF/pPoU7LE3NfZlt35\nf/yQL/7hw4T7y+t9rNlfya3Pr4vbd8Gfl/H9Z9aaryvqIj/4AeFIoYIsd7MTxgY9PbQx9nNoyYWy\neNMRLn3sE2rDpQJSfTHwO1/byORfLe5xI41dx2spvONN1uzvfuOnN9IrBN0V5SePTyRyO+xdMvW+\nYJTlDbr7xuPSMwQve+wTc7tVwJqs/tmx88kpW0s+un/3UGXEuv3zuzu44okVSb+Pel+gxS/61iM1\nbDua2FdvxFHbiXJ1Y1iswvusotXoDzJ5cDZDcj1xafKfl1Tyv/XRteW7y8ecbEcS276Wolb2xxTx\nSnUL/fnP9OS6njYXYIwcX1t7sJtb0jvpVYI+ZXC2rYXudkRSuK2TpoFgKM4X7FAF6c5IRQRj8s8a\n5fL/XljPpkPhCbqi+Qg0zlZ1y/d4bUTQf//Odj7aWdpi+49UNbLpUBW3PLeOSx5dHucCaQ2GSDfn\nEjGyKK0/9tqmAG6HgsuhmPsNLnzkY37w7FpCIY3CO97kNwu3dJvll2xNllgBt7O4//7xHh5arNe+\nj62r39Ms29ZidHwdMZLaeayGrz72ie3CL63FeMo9q5tpO1c9tTJqZbTuplcIulNVeO6Gmfzz2pPj\nfOgA/bLc5t8Oi4X+0uoSfvisLsQ3njGCv105HafFQoeI28X6uzhW08T8hz9i+c5S6D+JuvSBfEFZ\nDUTqyLSGG/+9ivkPf8SyHbr4x2Y5tgZDyJqzMI3rW0XfH9RwO1TcDiWhaBrJNk8s29NtLpekBT0J\nC/3u/23m4fd2EgxpxFaI6GmWbVvpiPdx/8KtrNxbbq7X2x4UY1H3HpqQ1hqWbD7Kh9uPc9frm7q7\nKSa9QtABZo7IJ8fjinKpGPTPimSSWi30Q1WNvLv1mH5MdhpfnNAfIErQjVj02MlTgMufWAFCcKjg\nLE5TNpBGU1KC/p9P97H1SDVLtx+n8I432RxehMHoPGwWYUoaQ6SDIY2PdtiPDgwLPFaUDQs9kWga\n20Oa1qxQaJrGFU+sYMnmo61ufywHKxsoqYi4Q5Lx7wMEQ8370KvqI/MHO47VmOGKBr0lDj3UIR1T\nx4mw8Zx7gZ5z3b/ikwrrmgKc99BS1h2o7IYW9SJBN7Crrhgt6PZv2VrEK91G0JuLuy7pN4d04eMM\n5XOOVscLeuy5v3h1I+f/cRkvhP2csdZ0e36DVuG64slo/73xFg0LPdZqTXOquBxq9ByBBaOdmhZ9\nbuyIwhcM8dHOUtsvfGuZff97nPbb96OunQwtuVzK6yNurdIaX1wn2lsyRTvCQjeeTUfkHhjfwVQf\nACV6FusPVLLjWC33L2x+EZ3OotcJup1gF1hqvSQqp2u10DyuiA/dsOSaczEcyplGleZhvmutaW1b\nWWoJIbR+EXI8TtvrBROIibUNiSyv5ixLo9MyLPTYY90OpVmXi1XorQK5PWaSNlm3SFswrv3Z3nLe\n/PxwwuNihexgRUP0fst7r/cF4iz0VI9yMegI15hiCnq7L2VO2vfUGkDJUt1gP59gvCtBO4bZ7eDE\nEPRkLHTLD1pVBI9fNQNo3uViEMDB+6FizlbWsmG/7md89IOd5v6r//4ZwZDGQ4u3RblkEgm6tTDU\n+1uP8cwKPb3eGkYZmwlpntuMZWkIumFRx7lcnPaTogZ1TVZBj5y77UjbBd0fDPGr1zdxuKqh5YPD\n1/7dom1c9tgnfO+ZNSzfZe9WihXkino/Ryz13q3tb/AH435+3RHlomkaL6w6kNS6t8nS3Pc2WYwJ\n4454IhELPbUF3ZqIZ8V4W+1xm7aHXifodpOiXnfE4k7scol+PW1oDqoi+CBsXWsapDkTl+J9JziD\nzFAVI5oGVn3cAAAgAElEQVQ2A/qEq5Xlu0p5+L2d/PTlSDapdSQQdT2LmFzzj8/42Sv6OdZSBHYL\ne+jnNmOhixgLPc6H3vykqDXKwdrGmsZoayVRh2DHsh3H+cfyvdz9+uakjvcHQzzyfqSzvPxx+7BQ\nO1eDGZlEdPvrmoLxLpdusNA3HKzi9pc+5/+9GJ9DYPDoBzv1yfgkSTTaaw2iA0VYaaUPfUNJFff8\nb3OPs+gb/IksdL2dsVFTXUWvE3Rngvrn5v4ES9LFDrnzvW6umjWM51bu50B5PcGQFrfgNOhWVTAU\nYmloMkHh4ByxilBIY1z/rKjjjGJRNRZRjBVCg0RiYrXcEllxzVmWSoyFHiv+poWeoFOwCnp0XH70\n8a2x0I1nYNcR25FMZxEMabaCsdUykrA+Y7t1VLvDQjdE4K0NRxIK2ANvb9Mn45OkIzom47fRgfOr\nSV/ror98xFMf74kr5dDdJJpjCUkLvWNx2iQWWelv8adbsRPr88YVENL04VUwpJmCaMUXDBEIadTi\n4UD2DM5TVlHvC8StlGQMfa0/1LIE6ex2XxZN06JcHgkt9GZ+wI4YH3qsFetWFdyqbqHbCUqtRdCt\nbqHYSdTmJi41TeNfn+zlUGUDoZDGD5/TrVGv28EvX9tI4R3NV69sSiKk0+4ZpDmVqGFylMvFF4xz\nP3XHpKi1TTuOtVwgLRk6xoceFvQ2Xsuubn6yFrdxy+4YMdU2BTiWoAxGR7iyOoN2C7oQQhVCrBVC\nvNERDWov1sSiN28+jSW3nRG1P5Gbw26pOmc4Yckf1AXO7pj6pqA5rN3X9yyGK0e5+O6neD0ms9L4\nYVl/E4nqk9h9eRv8wahhXlldE2W1TWiaxpMf7THdCclMiv7ohfUs31UaJ1r5Xjfu8DqadqJcl6D6\nZKz1FFu61sq+snp++dombn1+XVStkQy3g399si/heQaJOjIrdiKW4XLwya4yM7IoyuXiC8ZZ5O0J\nW/zjku0tdkx2WNtdlWRN+hav2RGhhuH/k40wiuXiR5cz5e7FQKRTaG2rumPE9JVHl3Pyfe/a7kvU\nURodVeyIv6voCAv9h0D3xOjYYHWpTBiYzah+mXHHXH1qId84eUjUNjvr2/C3+4Mhgppm6xer9wdN\ny+pwwRwAM8nISkTQLRZ6goxQO+uwpjEQJWYr95Qz/ddLePKjPdz7xmbmP/xRuK3R51q/eEb7a5oC\nfP+ZtXEdx9B8j1m50s61UWsZIVgFL95Cj7yubvTzajjN+2h1I9f84zNAH5JWWmLBMyyhos1ZgnvL\n6uK2xVp7dj/+dJfKntI6bv/v50B0p9ngC8S5n5IR9JdWl8RZcBsPVvHHJTtaPNcO63ejPROj1o6y\nI0Yaxte+rdFL6w9URspOJFG22o72dLB/fncHq/eVt3jcE8t2mx0PYJbYsGtrYkHX/+8mj0v7BF0I\nMRiYDzzRMc1pP8n0jL+6cAK/uWQyV84cZm6zi2Z0mqV2NZr8IdtjXl93iGBIw6EI/N4BrA+N4Dw1\nXtAjSTmRbaUJkpDsJvRqGv1Rgm5UdnxzQyR0LxjS4n7Axn1LKuqjBGNIbnrcscPyPLjDE792VnZz\nPvSXVpew8WCV+drgpy9v4Jbn17HlcDU/f2UDe0p1QR6U46HCEguuWkZWVnfO4Nz0qDb8+s1426E2\nJiXd7sdmTRaD6Gdc7wvGPfOWfPXldT5+9OJ6rv77Z1HbL/hzpMTvna9uNEsLJIP182iwcS0l6/Iw\nPgfoWJdLR4Sj2o1Uk6E99/79O9v5yl8/afG4X7+5haoGf5yA24UoJpogNjqeVPWh/xG4HehRMxbf\nmjWMp687pcXj7r14ovm3nQ/dKOp1rKaRF1eXUFobb1F/XlJJIKS7Y1yqYHFwBlOVnfxAfZnL1XeZ\nq6xglrIJV/kWCijHEYqI+OEE/jnjS2EtY1vTGIiy2oy2WIfmD7y9Nc7qNiZAT/vt+5TW+jh3XD9O\nH90HIUTcsQNz0unr1cskHKiILlYFMVEuUS6XID96cb0pZlYxNEIFqxv8+CyC1TfTHWWhW9tiFbZk\nBCnWPWEX6ZPujBH0YLSgx1qAdhOlVox2HWom3PLfn+7j4fd2JtwfS0sW+r8+2ZvUdezCM9/ZfDSp\nssh2mBZ6O7Nnj1Y38sE2PTO7tREzbU2QakuHVlrri/pOldqs4JVo5GOMDrvLQrd3KCeBEOIC4Jim\naauFEGc1c9wNwA0AQ4cObevtWsXdF01s+aAwqiISTngaLpdjNtmfBrVNAYIhfe1Sh6LwRmgm39be\n4v85X4o+8GP4UhpQBg1uFxV4qdQyqdC84b+9VJBJpeal764DhLRCvvPPrRQKLxVaJpV1TaYPO8Ol\nUtukf+GsCzp/sO04owuiXUyHqxr5zn8iIwaHooBDUBrwxbkm0pwq04flArB6bwXThubGvVeDf3y8\n1/z75TXRlfOs1pQxyqmoj7Z8NDQqGyIdpLUt1h9LMj/k2ExV45w0p0JjeGTltgj6Ux/t4Z439DDJ\nDJdKZYMv7gdqTEA3+oMcqmygf3Za1PyL0QFYz2vJjaBpGk0Bvf78gg9386Mvjo0KqY2NjY/lV/9L\nLrTT2qGGNI1gSOP6f61ieJ8M3v/RWUldw4qRJONvp4U+90/LzMJzyei59Xk253KpawqQ4baXskRZ\nz81x0v8tiXpdVutjZN/oYxJ1SIZh0l0+9DYLOjAbuFAIMQ9IA7KEEP/RNO0K60Gapi0AFgDMmDGj\nx00Nq0IQxN4/bgh6cxNLVQ1+00J3OhT2af2Z1rQANz7OGuJkb8kBckUtV07O5KMN2xmb6cdXW0Yu\nNeSIWvootfTX9pOj1JJDLarQYCWwEl6J1BQj9LyCz5nFTFc6jY4syuq9HHNmUNPkpVT1UokXt68P\neUePMk4cD3cUmTz87g5W7In4Dx2qQFUFTYH4yA6Afllp5GW4bH3Vy3dGijMttVlAwyBa0PVn+J3/\nrCYrzSJeQS3KQo8qFGax1pOxsBr9IR5buosrZg7D63aY57hUXdBVRUSVUP7zexEfd99MNxV1/qh7\nQqTzOv2B981ksL33z49rr3VkcfFfPm62nY9+sIsHF22jj9dNaW0TXxhfwKmj+pj7rS6V9vjQrZFA\ngaBmiprh7mqJJZuPsmJPGVfPHs6gnHRTWNtroVuriGpJTItaO6ZELpfV+8r5yl8/4e9Xn8Scon5x\n++3CHf+2dBfnT+zPsPyMZJrNsZr4kU0iQ8MwTAz78OZn17JqbznLf3pOUvdqL20WdE3Tfgr8FCBs\nof8oVsxTAVUREIxPLIKIGDX346pq8Os+dFXBabHym3AxbWIRQW9/Vu8r58WGHD4IDmGEM4PdgTr6\neF2U1vooyHKb9V8EITKp548XDmWQq4HfvLycXGrJFbWcOUQlV9Sxr+QAwxxN5DWWMULZRy41ZKjh\nEUQDsBLOtnQEDTtd/NIdGQ1kHO5LnSOLnQ0uhh0YzCVKI9VkUEcaHBwA7izGeuqorqrUzShLR9dc\nHXYr1h+itdRCdaM1MSlEbfh1ptsRJehRFrple7pTtbVcX1t3kMeX7WFfWT2/unC8eS23U4XGAIqI\nFnRrO/pmujlY0WBjoesTpYmKrRkC0+gPcd9bW/jZvHGsb2HN0xdW6RE2RnRTrJ++JQs9WazXDYY0\ncz4kGaNR0zSzBs+SLcd4/0dnmQlobfFjJ/L7JxOFaP3dJbLQPwlXgFyxpzyBoEc/x/I6H79ZuJWn\nV+znw9vntNwI4J7/beb7z6xl491fNEdUVkNjRJ9IxxBpp/6wjWi33y3axo++ODap+7WH9ljovQKH\nKsBvn9llRHw0VwfaaqE7YnqF7HQng3PT+XS3xgfbjkddyygAluFyAPoPXEOhGi/V6UNpUBU+CEX8\ns/vS+jG6IJOnDuzh7CH9eHvTEXOfGx/Z1DExN8Cl4zy8/ukmckUNudSSI2rN0UCuqGVg02489dXM\nDFaj7g1xlsvS4Mf/D4BnAWqAuwW4vHzqdlCrpVNLGnVaOnWkUUs6tcbf4f/rSINNfvIONzJNHKaW\ndLZvO042+v6A5evmD2nU+gK4HArpLjWmlK+9hd4vy82+Mt23n+l2mElaZeH5hGdX7uf1dQd59oaZ\nQGQiVBd01faafbxuNhysMjuOr580hHUHKqltCsQJ7s5jNWbUlHXfgg9387N542iJ2CzlWNG2tuvB\nRdt4YdUBlv44OdGxEiXommZa1sk4Ae6wrItr9MXGs2lNBrBBXYK5iGTCKWfdHwkZTGQRG4ECjy3d\nxW3njYla7AbiBd1wlewvr+eR93bw/bNHt9iOY+FOvbzWFyfow/I9ZKbrJTye+mgPHyfI4n3k/Z2p\nI+iapn0AfNAR1+pqjEzFrPT4uipGklJ9Amspx+OkusGPP6D70GOzULPSnTgUEfWlrg/7Zj3hRTQy\nw26IkX0z2HVcHxIHQhrlddHDvNqmAPW+AB6XSlZ69MfWhItjuHivAt5bDnBywvf7jeKhuB0Kr6zZ\nz1XT8ln02WZcgRq8opHnvjURmmp4cfkWjpWW8r1T+1NbU8nSz7aTIRrw0kiGaCCPGjJowKs0kEEj\nbmH50b64gLOJHiUYNGpOakmnTkvDuS0Ln+rhNIeDpqAH7/4cRjqC1JGOZ9UW/Nm5qOlZnK5to0Kk\nUUsa/UQf6ghRgZdRBTms3a+XKLU+3zpfkM/26hFAQ/M87Curp8EfjLLQrfTNdNPoD1HTFKCP18X9\nX5nMNxZ8yrIdpXELjZz70Icsu30OGW4Hv317a8JnnAiXmpzYGBidF7RuwQ2r3zgYClksdMHOY7U8\n+v5OHrh0cpwBAvB8eBQBMCTPA0Q62JYmiu2wJsNZsbO4j1U3cvNza3nk8mn08bqj3CWJ/PfWyK+F\nGw9zUfGgqP3Wa+gJTpF9v1u8PSlBN7DafFa3nlGq2ZiXAb0z7I5yBSe8hT6mwMv2o7VMGZwTty/W\n5ZLjcZp+379dOZ09pXXcv3ArJRUNug/d8gPJz3Axa0Q+60sqo75ENTEW+oDsdK47fQSnDM8zkxju\nem0jF0weGNWWRn+Qel8Qj1MlM82+qFcyOFXd/dAYgFqRwWFlACMGj2bK8DwYq1uZuw5u5ckDuxnR\ndyrfXbwGOKv5axLQBV408tEPZ/Dqim28smIbGTSQIRrx0oDX8neGaGS0A9yhevqKKjzaEbJrmpio\n1uEVjbD8ZfPaj6mAYVzXos/WALXHvRxzeSknC3EwnzkONxVkUqZl4diwnjlKiNMyxrJX1FCuZeFO\nkEFsRPUcr2kyk9I+2a0P4//24a644yvqffxm4RZzMZLW4HQ0b6HbWaGh8IR9rFGxaNMRs35/LNaQ\nU6sPXQDX/uMz9pfXc9OcUYzq5222vb5AiAZL0lWZTZRXSyTqBOwSxJ76eC+f7i7n+c8O8L05o6Lb\nkqBDs7pl7KJ4rEXs/EGt1VEvTlXY5jUYna/LoWDXNCFo18pjbeWEF/TnbphFgz9omwVq+H/fCy+C\n8ejl08w6GqeOzDdL636yu4zCfI8p6H28blb94lwgcW0Zwx3gcih8aUq0eNf5glGWklMV1PuCNPiC\npLtU06pvCw4lXCI3GCIQ1HCogte+NzvqmKL+mfiDGt99ek3c+TeeMYK/fbjbfH3vRRO487VNVJJJ\npZYJ/SdywOtmaci+xIJJaeReIU1jZF8vCzceQRDinvMLeeTtdRSk+RG+Wq6YmscXRnmpqijn8XfX\nk0cN43L8+KuPkUsNg/xHOUutII9qXCIIR+FbLmArfDs8UghsdHGzW48YKtMyTfGfcWQkR9UG0sv6\nkYMXjg5giKuGQz4P//l0v23TrYtrWzF+/D85vyjOgj9a3cj6mEUPYifs7PzNDf4gGW5HnNvvF69u\nTCzosT70QMSHbqydmqimkZXlu8oY98u3mT0qH4DjCTKbmyPRXEC1TSasYdFuP1qTVLIYRI/O7OY7\nrKOgQCjU6hICuR6X6XKJmucJRQS9vinITptSDYer2hYm2h5OeEHPy3Al3BcbemQdojpVJcpNo/vQ\n9eOtE4F2HQVEC3pLZKe7qPcFwy4XR1SoW2txqgK3U0XT9DhpOyYOyk54/qTBkX3Lbp/DgOw07nwt\nsgRXTaM/ztc6bkAWW2zqxIOe8t/oj/wgNBTufHs/kMfAfrpb5ezcMWRPHc2xozX8e3EBALNz8/m4\nTLekx+TpoyzQ8NJAcX6Q2vKj3H1eAf9+dw251HD2UIX9B0rIE9XkiRoGc5w8pYbsHYuY5AQM78Zf\n72SZAqRBpZZBuZZJOVlmR9B/5TK+WO1noOKgjEx9O1k01VUyJCedCYNzKMz3xL1PY9FmK/e+sZmv\nnTTE/DwNkXjoq1O47YX1gG7J2gm6Xd6EgS/Gh25Y6FZRbE0qvdHxxArmZ3vLKav1MbrAy8i+9tZ+\noqJatoIe/v+1dYf44TnRrpBEk6JWCz22fRV1PkrKI/NQ/mB84p1BovLNeRkRQbeOoIzO16kqBLUA\n5z60NOo8gTAFffLg7LhFyDuLE17QW4NTFcwelc/HO8twKCKqnnlIi/hIrZUDreI+f/IAc1GG9HBM\ns1Xvn7p6Btf+I36Vn+x0B5X1eqZoukuN8wf/8JzR/Ond5NLN05xqnC83lpF9M+ib6ba1eMZY4tzz\nva44P+z3n1nL2P6RY9KdKs9efwrvbD7Kj1/6PO56GeEww02H4gXf8JEbz9M6sZmTHumII35aQS0e\nNje6KNcyqR82k5eCelvSho/iz3vik3xe++7JXPfXxeSJaqbkBXlg7kCoL+Oh1z41xT+PagaJUiYp\nu+mz8WOuDvkh1g548BYW4qRhVw7K0T7826mao4D6JRs4+H4p5yhZlGnZlJJNqZZFI26++finvPq9\n2QghTJGwukIMC7M2xhedwE4AYn3omm3Wr2Gp+gIhTvq/Jdxz0QQujBkpGhjiW1rbZLqAjlY3ctlj\nn5jtXXLbmYC+8PbEQdmcVJgX1f5Y7GrVWEU7duWvRIJudakci/m+nnLfu1GumkAw3kLfeLCKDQer\nEsaV53sjH7Sdhe52KLYjq5CmmZ3EqH7eqEqfnYkU9FbgVBUWXDmD/eX1OFSFbIuF3ugPmsJjtcpV\ni7h/afJAU9A9zkgEhsGsEZGYZCvZ6U4OVTZS7wvSx+uKsupvO28MN58zmu+fPYrRP1/Y4nvwuFQz\nvR/gipnxyV5CCK6cOYyH3tlubrv61EJuPHNElP/ertDZ0u3Ho2b605wKOR6XbUgZ6Ik9jS3EXBvP\n01qP/vJThtI/O40nP9pDjaV2SabbYfourW6FtJhMUYO+2V6Ok8NxLYe09GyYeBoAj7wy0ExPv/OC\n8dwbnvC698Lx/Pb1VeSKGvKp4aopXpZv2MbPzuzHG59uYFymn+GeBjzlBxjMcfKVGjwfLeK3NtMe\ntVoapceyOfC7vmypTqNo4BBudcDQnXuYpxylVMsmcHQgftcwth+OHtIbiXB1TQF++domfnD2KArD\n4XNNgZCZVKX70G0EPWypVtbrWZH3vrGFeZMGAHDVrGHsK6s3cw2MzFN/UKOqwU9uhiuq9LP1+d8d\nTn7ae/98bnthXcKQ3zpfEF8gxDMr9jGmfyZDcj1ReQlldbGCHhHNYzWN9PW6EUJE+cRjDZBYv3sg\nFL8WrpHdHDsiMMj1RATdLoPZpSq2cx+BkMbR6kZURTA0z4MvECIQDNlORHckUtBbgVNVyHA7GDdA\nr3UeK+jGMNgq6FYL3WpZG5OiVrdOIvdMjsdFgz9InS/AUJcnStDnTuxvtu2SaYPisjaH98mISihJ\nd6lRP8bzxtv7YWPbMn1YLgOy022PjcX6BTfmFaw/DCtet6PFyoLGM7Ra6G6Hwk1njeTJj/ZExZX3\ny3JTc1x/3S8z4sdP5FqzWmBW0V/9i/OYeu87AORYPuc7X98MeKjVPByggAsHj+eldf25esJp/OHT\nFXypcCBfGN8/aj3Xu+eO5LGFK+kjqrh4tJOtu3bRh2ryRRV9RBX51dUMFUcZVLqT76tVqB+8wqNG\ns56/F4CLNZXT3dmUaVmUatk0+PNg8fu8s8NP6CCsZhyFp08Fbz98Ph8ZLgeNfl/Yhx4vqoa1acSY\nq0pEpPpnp3HIUmq4pilgjtiO1zbx6e4yc37FiEZZs78iqlBZeZ0v6rvoUESc8C3ceDgq+3Xa0Ehg\nQkVY3H998UR+8epG/vPpPi6dPpj9ZfWc8eD7/HRuETeeOTKqJkxLPn5/eN7IDjuXGOjBDQaBUIja\npgAOJdKROFXF1p0SCOnlrjNcqulSq/cHyZKC3nOIXYTBKuhuh2rG1kb70BXLMYqZSGQcY3WFJpqo\nMu5TUecj3aVGRdOoCToMgHOK+vHwN6Yy4a5F5rY0p0p+RiSmMNsmXBOIyuq878uToiZuh/fJSFjL\nPRajrYk6q76Z7rihcixGp2cdWaiKsJ1/KMhKY9fxOoSIrn2fSNDdDn2SuaYxECXouRku3OHl+BIt\nFQiR59fg1y1Op6rEfU/uWrgLyOewls/MguG8uH2IzZXg/NH9WbzpEKtvm8a+/ft44OVl3HlWX15c\nukYXfqr1/0UVRcFDaCuWc3GwiYtdwObwP+BJoFzzUurKpuz1bEIZfbnLkUZp2N1TpmXx3CtH6Pel\n2TS69QnPo9VNpjVtlLGwMmFgFh9sO87xmiYeWBQpODYs38PqfRVc8ujyqOPX7q+Iep2Z5jBF2iB2\ngZfPLYlZxpyB8XzXhSeUDTfGki1HufHMkVGTp5X1fpoCwaiO38pr6w5x6sh8231HEtS5ybP8VvxB\njYl3LWJQTjrfDI9sE82BBcPi73U7TOOtwRckqx0RaskgBb0VxEaseFwqUwZns76kigy3aoYnWt0S\nVnF3ORTuuWgitz6/jj6Z+hfFqnOJ6j+Ygl7vx+OK9oE7ojqM6C9yXoYrrsZFulNl3qSIVZ5I0L92\n0lBzsjPDHX3dRbeckVTqNrS8EtGofl7bCAErZuanI3pS2u7H1C/8XPtluqP25zcz+V2QlUZNY23c\nEoMORdBE4rVfIfL86sPhfU5VaTaCJCdmpJLrcZpCpyqCEAqKtx+iIIPloQquXuXmaLAg7jqF+R7e\nuvk0Tr7rVfqIKq6Y6OG6qV427tjFOys3kC908e8jquhXv51JahVZwmJJVgD/0v/c5HZTqmVT80g+\nL7qCDFnj5aymEFc6/QRR0BAMrvJyhbMBnnPxa6FQ6QyioTCwIYN9zkZCKIQ0QRCFEIJhn7zKPY4K\n83yP4qLKESSEYh4zcftSblbLzGOCKJw9sT+LtxxnzJ5VXKWWMXrfNi5XSwghCKwuo6CikS8ruxlS\n54WNhzmpfie5SgMBVIKoNG1zsXB7GQPyvEwVO8Pb9Xu+vPggZ3xjOoPFMYKaSgCFIJH/g5bXWrhu\nYWZMuQrQF7yxTorGMjA7DX9Q04MY3A4zACKZWv7tRQp6C4zom8HucMJP7GpIQgjuvGA8lz72CV63\ng6L+mdxy7mi+dlLEAlNjBP2LE/qz+Z7z+efyvfo1ksjfs0bTpLvUKKGy+uiTiZjxuNSojiMngaC7\nHAqnj+7Dsh2lcR1FMvcxsH7hH7tielShsB99YQwXFw/i3XBYaCKsiTEGqiJsQ0KLh+Tw6rpDZke2\n+NYzqKjz2SaOGfTLdLPzWG1cRUbjs0vU6QFkh8W+wRfAFwzhcihRo7K448PXUgQ8dfVJ/Or1Taag\nGxNzqhoZfcRODoJeUri6MYAvqK+UVat5WK8MZAkD2Zo5jj8Fx3DB5AG88fnhqPNc+Mm3uHqKvI18\nc6KHRSs/J19UM9hXT5PmQxMqCkGcIkAaIRQ08kWIelGL6tPwOAV9hV/f7nOQJ5pQCSEUDZUQKiG8\nhxUuUH1haQzhCIBQgyhCP0ZoIdTdGsWxj3YnnOIE9sLZTmAN3Gcc8z8oBP7gQs9kfkkv9Ro1Qf0i\nXBz+8xWb5DZeho/stscQ0gQBFJT3nFzm1jsbz0tuVrpDBFDIWJHGfFeQjF1pXO8KEUQhgIrqcOLU\nHPhLFUS5g6aQYMzaXJYMDJDtGweMaPnm7UAKegu89//OMlefiR2GQmSyJjfDhRCCW84dE7XfOntu\nFUZje3PRCgZDLDXBM1yOKEFN5KNPhHHuNbML+fvHe5OKaXcnWBzb4Bfzx9nWKYdoQT9/YrS//trT\nhqMoosWoG+uCGZHrCtsKmV+eOpiFG49wx9wiIBKVY3URPXbFNL7zn0iMfUGW7pqJnTg12h67/Rsn\nD+XZlXqMuiHQhh/fpYqozyQW41oXFQ/irLH9yE6PTDwbk5eqEHF14K143Q6OVjdGhQSW1zWZNVgA\nfvzFsXGC7sPJ4bDrBw12KOnMKprEr5evBCBLdVDtD/DrUyayZn9FlA/8lS+fyoOLtrF8VxmzBueb\nyVd/+PIUbn1+fVwbvzNzJI8tjSRmnTI8jxV7yrn61ELOLurHVU+t5MbTh7Ng2S7UcKcxZZCXX8wr\n4orHP+ErUwfy2tr9PHfdyZSU1/Lzl9fz72tPorSmnp+8tJ7RfdJ58qppfOdfKzhQWqN3GgT5w2UT\nueOltagEyXELGpp8fPOkgbz02T5UQnz/rEIe/2AHqgjiIIRKEAdBVELceNpQ/v7RLn270Ld/YXQf\nlm49jEqIcwbl8dH2I6hoTMzysKO+kpHpaZTU15jXKsr1UFFbj6YFUUNNZBLC6w8xyhmEdHtXUEci\nBb0V2A2lTyrM5epTC/nOmSNtz7EOs6xCbK5sEuNmeeArk81VdQysceEZbkeUcEeNAGKE0ehKJg3K\nZkN40QPj+Dvnj+f2LxYlNeveUp9z3ekjeHzZbltrsjn3Q1q4g2tR0G2iNBJZwdkeJ8/fOCtue77X\nzXM3zGT8wKy4hUX6ZekmW2yHaDyrWP//5MHZPLsyfL+woD+zQhd4l0OxHYYbGM/DmCD0WjpUI8RP\nUSDT5eSGM0awwJLEZWCEwVknkw9WRCYxhWg+v8LAFwxFRaEYnZLuQ49+z5lpTk4ensfyXWVRGa+D\nc+o8XYAAABQSSURBVONj7oEoMVcVEZV3YTyf6ibdbRMIuzc8GVmoaZnU4KE0mE4FWahZ/chQcjhC\nCceUvlS7/OzXjuASXug7lu3aYXZrkUn/43nT+DS85kCOcHLhKQNRivqxcIW+EMlXhp3Ef0PRi5KA\nPiFbf3Ixjy79IGr7yOLp/HqjPqp0j53EzzbrtW5uGjmSBQd3c8WIYfzjyF7z+PtPmcQ7m49ypLqR\nkAaDctJ54lszbJ9RZ9DrFonuTOx+qA5V4VcXTki4+LQ1IcQqGIaFHus2/+pJ8RNmRk0NgAHZabjU\nSE8f66O3438/OM2MIDDeg6IIc7KmJZLxlhuz/pdMja6l0Zy4GRZ2Swt724XdtXZkAjBzRD5Zac64\nNhWEo2Fi72OE8cWGZ55vydA0LG5j0s5uUtSK0TkYtVmsk2TG/Y2RYFbM6MmlKrx9y+nMHKFP7BlV\nG8cNyGKvpe5LdrozqeQzfzAUtU6ttY2xHX1WmiNcSC66RonHpfLFCfE+fivpTtW8ntuh4Ap/3tZw\nR9DnE4zP0vjduFTVnP8orW0yM08DwRC+QCguwuQPllDbJn8oLu8imSX5Mi3PLtcyf/KzVyKFy6ob\n/SiKiOvsHapCmkul0R+krimA1935VrkVKeitoKUJPjusxYnsLHS7Ko9PfmsGv79sSuQ8yxdyUE56\ntA+9GWGz5koYgtucO6A9GBbnD8+NjueNvd8v5sdXJWxO9MHeQjeewZLbzmDlz85tVVtjP0djgro6\nRmB+MX8cK392TpwP3TrRHOt3h8TlHiAi1oar7teWVbOM92k8stiaPWlOhaL+WaabzBD06cOi6xDl\nelxxIz87F44vELJ1lzhs3EaZaU7TALC6EV2qws0JYrgNzhwbWR3CpSqmQRK7dGCOx2V+F+rCIwen\nQ5AfrrdTXuczBd0f1PjmE5/GhUIu3xWp2W8UZbO+l0QJSkII8/cyMCfyrPpl2RtqR6qabEcyAj23\nwszsbkdWd1uQgt4KmvuhJsJanMgqXEaUiJ28njOugK9MH2w5L3LUwJz0qNdWv74rZvLSGoliWJuD\ncpKLJYdIxmKiGHIrRocRK3BThkSLzXWnx08KfTls1b9z6xk8cvnUuP1WMXrmulP49cUTzWiWUf0y\nzYnJZImdCzEEOzYe3qEq5g/6sSumm9udzUxE7z5e12xp2IjLRRcWQ6wgsgC2IcaxVTWNVZeMCV4j\nkSY2P+AKy1q5BvleNzv+by7Xzh5ubksUdaEqSpyhkeZUzGgna7ihQ1UY2dfLMJtyBwPDo9biwTmm\nVe92KuaIrDYmbDEvI5I0Z1joTlUhJ92JIvTiYIaLKBAKmVU1m8PtUKJGG82tbTAs38Mdc4t46pqT\nzG3G9yyWfWV1qELEdZwa+oiurilgFtPrSqSgtwK7SbiWuOXcMZw+ug/P3zAzytK7qHgQw/I9XDWr\nMOG5U8NuEiEEz1x/ChcXD4zLFG3OQrdywxkj2Hj3FxNaHHbcMbeIf1xzEsVD4itRxhI0U6EjX+D/\nfvdUfmxTA3rhD0/n8asifsXJg3PYe/98RhdkxlWZHNXPGzU/ceqoPlwxc1jcDynT7Ug6+ibWZ28M\n6e0mvQ2sE7rNLS92zezCZj8HI2yxIDP+c4iNy46NWTbanWVa6HpGbKzonDI8L+7a7rDv+tunD4/b\nF4tDEXHFsYQQpuvJuri3UxWkOVUW3XJG3HWM+QGPJaRXt9D19xH7fnM9EXeY0dk4VQVFEeRluCir\n85nzDK2pRWPtgBOVwhbh9/idM0dGGT2JlrbbW1aHqgoaYqpJappGhlu30JsCoRYDCjoaOSmaBN89\nayR//SC+lGoy9M9O49/fjl+wuiArrcXFC/797VPM7LtTR/bh1JF6aYBEUS7NRUYIIVpd1MvtUDlr\nrH3KfizpTv1L7HIozBqRz8wR+eb6pLGMG5BlZtvasfTHZ/HG54d5cNE2RvfzJjVxu+rOc5NapxLi\nXS4TBmbx07lFXBzj/48l0+3g2tOaF8QR4SJVr31vNoX5Gfzrk7383uLXnT4slz99vZhzx0X8zm/d\nfDrzHl4Wd63YUEujozBcMYbLpW+MoBsug+V3nM0fl2znhVUl5qSkM8YoMfIorChCRC3obWD40Cst\n1SZdFt94LMZozet2mKMWlyOSGBe7AlaOJ2Ks1Jo+dP11XoaLI1UN5GUYORnxpWmvPrWQf4TDgQ0O\nVzVGddSbDja/qpQdsXWNFKF3KJX1fg6GM2ozXCp1viCaplvoZjVGVVroPY6fnF8UtZ5kV+F1O0yB\nsOK2fEmsowZrJUQgudnMDuL5G2dyx9wi0l0qz94wM86X3hqG5WeYE3/JVqlzO9SE9VpiiRV0IQQ3\nnjnSDF9MxIa7v8it50XCUo0ELbvswylDcsj2OLlsRmSSe3L487moeFCU5ZcodDTWQjdGKsZ2o6RD\nH2+0oBsTeQNz0s2wzSHhaJTYztHu++VQhDlp+4OzR/H2LacDkXIVNU3xbkS7UYuxz+NymCn6boeS\ncAST63GZAm4IqGFd98tM4/1tx1m4QV+py67zvtUSMnz1qYWWdkTa9lyCFP/mlud79/+dGfXa6rYv\nCUcXGYELGhoZlmCD1uRsdATSQk9BEn1JPC4Hd31pPDuP1fL0Cvta3p3FqH6Z5vJsHcGYAl1oLi5u\n3mpuC8ZcyDkJCoYlw+775pki8MS3ZjD+l4tsjzPEa8qQnLi68waJyiJYhd5qUBjbV+/TfcjWjuyp\nq2dEiasxKT86/DxjOzO7kFFVFebk4fA+GRT110dTdiO82EU7rBh1YjJcqunCsQvrzEpzUN0YICs9\n3m1mPJtrZhfy0c5SdpfW4QlPOsZidW9cOn0wI/t5uXDyQMptrPlYYhP8nrn+FLMTtM6dffSTOZz2\n2/fN18P7ZLD1SA3D8j1sPVKjW+iW55RM3fmORAp6CtJcr3/N7OH8d3UJT6/Y35UGeoeTmeZk133z\nEopde1AUwbLb58S5Klp7DYN0p8o5Rf3M+h5WcjNcPPyNqcwaYV9DBBJHTyXKbvW4VFRLgSirxRub\n1fvt04ejCD0ZCuIn9h2qYMltZ7LpUBU/fG6dvk0RZqVC63fNboLQLmrqri+Nx6kqZmy+x+2I+NAd\nSlwn0MfrproxYL4vK0bnZJ3HuXjqIPPaVqydk9ft4Mrw5HAygh4bnWC4NyH684mNu3/wsil896yR\nPG1ZDMXa8SUbUttRSJdLCtKSyNmV8U1FOrP9Q/I8SbtoWkIIwZNXn8TZRfbx2BdOGdhs55Eoeioj\nQZ6AECLKerdaprEC4nU7+ME5o02rOLbzGDcgi1H9vHzJMhmtKgJfwAhzjVwvx+OMMybsQk5PC09c\nG1Z+mlMxQx3dDiVKeN+8+TQe+lox540vYFh+hu37hehEqZkxneOkQdk8c/0pUZ2sxxL/HWzlKkWx\nGJ2WMe9h/V563Q4mD85hRqE+XzS6wGvOV4B0uUg6gLkTB7D21MqENZ4lPYtEFnpz0TRWUbRa5S0J\niNWifuk7s8yJa6sYOhTFFGOXJelLCEFBlpsDllWA7Dpdw58cCEU6BWv9cKsbYsJAfV7BGvX0wo2z\n+OrfPom6phCCa2cPZ1Q/LyP7RoT/kcunMnfigLh2ZLisyUEth902hxCCT356ttmpvPzdU7nkr8uj\nFgS5dPpgZo3MZ3Cuh61HIou1SEGXJMUjl09lVYI4XJdDz16VpAZWK/edW+PD/+ww4sQvnT6Y7HSn\nuYB5otKxBtZOYkZhfHgj6CJtxMnbZdVaBd3KvRdN0DMlwyOfmSPy2VNaR67HGVXqormOCuBkm7BL\ngF9+aXzctsL8DNtOxZoPke9188jlU/n+M2sT3rOlsaA11n/KkBx23Tcv+nwhTHfMKMtEc1dHuUhB\nT1EumDwwLmZbkppYBWl0QfTE8k1njYwq/RB7juGbz07XBT1ZN1WihBnQrXh/wL48bHOlhK+Myam4\n+8IJXH/6cPK9bjPJraPcaINz0ympaEjoo47NGenfQgRTC31Mq7BGEkkLXSI5wWiuHMPt5xfZbs/x\nODlY2WAm75w2qg/7yvYnNQn38k2nMsymkzBQLevlxka2GOGWXxhfwPzJA5q9j8uhmGGRhsulo4Tz\n6lML+fWbW8hNoggZkHARa4Nkyli3BpeqmOWUuxIp6BJJN9OSC8IOw59rLHv4qwsn8NUZQ2yt+Vim\nDbVP+DJQFcEDl07mzA2HmTAwOgHMsNinDs3lolaElBqx20bfde9FExg/MHFy2bLb5zS7NOG3TxvO\n5acMtV3X1o7cDBf3XjSBrHSnGc3z2vdms+1oDbfbLF7eXtyOsKB38pJzscgoF4kkBZkaDuMzLECn\nqsTVzWkrDkWQ43HxzVPiSywYr1pbRfCscIGu/mFf9JWzCpk+zN5XDvrEqrVsdCzWUgTJcuWsQi4q\nHvT/2zvfGKmuMg4/P5ZdsLTuQkHcAC2g2xqsCoQgWCS1jasQU/uhiUtMJFXTRE1aUqOBNmli/KQf\njDUxto3/+kFrtVolREVsm/jnA3X5Vyi4slVMIdBdbdoaP9Ht64f7zjI7zu7szN6ZOXfyPslkzj33\nzr3PbM6+c+57zz2Xfp9j5n2r+rjlhszrE1VmOZ0LpZFHPTVmEs2b6KEHQQG557YBblrRywcHltbe\nuE5mk+euN5Vwz60D7Np8Xc27cVvBb+/dPtn7f9tbFzblLvAFk3P9x0XRIAhqML9rHoPvfnvtDeug\nu0tcnrAZJylrlHnz1JJg/ocvfWjK9L7V6L2qu+4ZOuuldC2j1lz/eRMBPQgS4KGh9QzkOHVCI3R3\nzePyxMSU59RWkudokGZwXZVpfNtB6Qxm4s3W3q8dOfQgSICPr18x40XCVjB5N+kMKZetPhHZDcvb\n++OTOu/x/H9edyPPluihB0EA1H5yFGQP4f7AO5YmkQtPma/ecRN3bFhRc7hk3kQPPQgCAHr9CUmV\nj3WrJIJ5bRZ2d3HzO/O/YF2L6KEHQQDAD+/azFPHLkw+Oi4oHhHQgyAAsrHftR74HKRNpFyCIAg6\nhAjoQRAEHULDAV3SKknPSjot6QVJ9+YpFgRBENTHXHLobwBfNLOjkq4Bjkg6ZGanc3ILgiAI6qDh\nHrqZXTSzo17+D3AGyP+JvkEQBMGsyCWHLmk1sAE4XGXd3ZKGJQ2Pj4/ncbggCIKgCnMO6JKuBn4O\n7DGz1yvXm9mjZrbJzDYtW7ZsrocLgiAIpmFOAV1SN1kw/5GZ/SIfpSAIgqARZDWmmpz2g9nM948B\nr5jZnll+Zhz4Z0MHhKXAvxr8bDsokm+4No8i+RbJFYrlO1fX682sZopjLgF9G/BH4CTwplffb2a/\nbmiHtY83bGabmrHvZlAk33BtHkXyLZIrFMu3Va4ND1s0sz9Bzk9WDYIgCBom7hQNgiDoEIoU0B9t\nt0CdFMk3XJtHkXyL5ArF8m2Ja8M59CAIgiAtitRDD4IgCGagEAFd0kcljUgalbS3TQ7flzQm6VRZ\n3RJJhySd9ffFXi9J33Lf5yVtLPvMbt/+rKTdTXKtOnFawr4LJT0n6YT7fsXr10g67F5PSOrx+gW+\nPOrrV5fta5/Xj0j6SDN8/Thdko5JOlAA13OSTko6LmnY61JtC32SnpT0V0lnJG1N2PVG/5uWXq9L\n2tNWXzNL+gV0AS8Ca4Ee4ASwrg0e24GNwKmyuq8De728F/ial3cCvyEbBbQFOOz1S4C/+/tiLy9u\ngms/sNHL1wB/A9Yl7Cvgai93k00hsQX4KTDk9Q8Dn/Py54GHvTwEPOHldd4+FgBrvN10Nak93Af8\nGDjgyym7ngOWVtSl2hYeAz7r5R6gL1XXCu8u4BJwfTt9m/YFc/xDbQUOli3vA/a1yWU1UwP6CNDv\n5X5gxMuPALsqtwN2AY+U1U/ZronevwI+XARf4CrgKPB+shsx5le2A+AgsNXL8307VbaN8u1ydlwJ\nPA3cChzwYyfp6vs+x/8H9OTaAtAL/AO/tpeyaxX3QeDP7fYtQsplBfBS2fJ50pnVcbmZXfTyJWC5\nl6dzbvl30dSJ05L19RTGcWAMOETWY33VzN6ocuxJL1//GnBtC32/CXyZKzfUXZuwK4ABv5N0RNLd\nXpdiW1gDjAM/8HTWdyUtStS1kiHgcS+3zbcIAb0QWPbTmtSQIc0wcVpqvmY2YWbryXq/m4F3tVmp\nKpI+BoyZ2ZF2u9TBNjPbCOwAviBpe/nKhNrCfLK05nfMbAPwX7KUxSQJuU7i10tuB35Wua7VvkUI\n6BeAVWXLK70uBV6W1A/g72NeP51zy76Lqk+clqxvCTN7FXiWLG3RJ6l0N3P5sSe9fH0v8O8W+d4M\n3C7pHPATsrTLQ4m6AmBmF/x9DHiK7AczxbZwHjhvZqVpuJ8kC/ApupazAzhqZi/7ctt8ixDQ/wIM\n+CiCHrJTm/1tdiqxHyhdkd5Nlqsu1X/Kr2pvAV7zU7CDwKCkxX7le9DrckWSgO8BZ8zsGwXwXSap\nz8tvIcv3nyEL7HdO41v6HncCz3hPaD8w5CNL1gADwHN5uprZPjNbaWarydriM2b2yRRdASQtUvZE\nMTx9MQicIsG2YGaXgJck3ehVtwGnU3StYBdX0i0lr/b4NvNCQY4XHHaSjdR4EXigTQ6PAxeBy2Q9\nic+Q5UKfBs4CvweW+LYCvu2+J4FNZfv5NDDqr7ua5LqN7DTveeC4v3Ym7Pte4Jj7ngIe9Pq1ZEFu\nlOx0doHXL/TlUV+/tmxfD/j3GAF2NLlN3MKVUS5JurrXCX+9UPr/SbgtrAeGvS38kmzUR5KufpxF\nZGdcvWV1bfONO0WDIAg6hCKkXIIgCIJZEAE9CIKgQ4iAHgRB0CFEQA+CIOgQIqAHQRB0CBHQgyAI\nOoQI6EEQBB1CBPQgCIIO4X9iGnorp+WAJQAAAABJRU5ErkJggg==\n",
"text/plain": [
"<matplotlib.figure.Figure at 0x119bae4d0>"
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
" \u003cMovieInfo id(1), title(Toy Story ), categories(['Animation', \"Children's\", 'Comedy'])\u003e\n",
"\n",
"\n",
"这表示，电影的id是1，标题是《Toy Story》，该电影被分为到三个类别中。这三个类别是动画，儿童，喜剧。\n",
"\n",
"\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"<matplotlib.figure.Figure at 0x119bae4d0>"
"cell_type": "code",
"metadata": {
"editable": true
},
"source": [
"user_info = paddle.dataset.movielens.user_info()\n",
"print user_info.values()[0]\n"
],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n"
]
}
],
"execution_count": 1
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
" \u003cUserInfo id(1), gender(F), age(1), job(10)\u003e\n",
"\n",
"\n",
"这表示，该用户ID是1，女性，年龄比18岁还年轻。职业ID是10。\n",
"\n",
"\n",
"其中，年龄使用下列分布\n",
"* 1: \"Under 18\"\n",
"* 18: \"18-24\"\n",
"* 25: \"25-34\"\n",
"* 35: \"35-44\"\n",
"* 45: \"45-49\"\n",
"* 50: \"50-55\"\n",
"* 56: \"56+\"\n",
"\n",
"职业是从下面几种选项里面选则得出:\n",
"* 0: \"other\" or not specified\n",
"* 1: \"academic/educator\"\n",
"* 2: \"artist\"\n",
"* 3: \"clerical/admin\"\n",
"* 4: \"college/grad student\"\n",
"* 5: \"customer service\"\n",
"* 6: \"doctor/health care\"\n",
"* 7: \"executive/managerial\"\n",
"* 8: \"farmer\"\n",
"* 9: \"homemaker\"\n",
"* 10: \"K-12 student\"\n",
"* 11: \"lawyer\"\n",
"* 12: \"programmer\"\n",
"* 13: \"retired\"\n",
"* 14: \"sales/marketing\"\n",
"* 15: \"scientist\"\n",
"* 16: \"self-employed\"\n",
"* 17: \"technician/engineer\"\n",
"* 18: \"tradesman/craftsman\"\n",
"* 19: \"unemployed\"\n",
"* 20: \"writer\"\n",
"\n",
"而对于每一条训练/测试数据，均为 \u003c用户特征\u003e + \u003c电影特征\u003e + 评分。\n",
"\n",
"例如，我们获得第一条训练数据:\n",
"\n",
"\n"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"%matplotlib inline\n",
"\n",
"import matplotlib.pyplot as plt\n",
"from IPython import display\n",
"import cPickle\n",
"\n",
"feeding = {\n",
" 'user_id': 0,\n",
" 'gender_id': 1,\n",
" 'age_id': 2,\n",
" 'job_id': 3,\n",
" 'movie_id': 4,\n",
" 'category_id': 5,\n",
" 'movie_title': 6,\n",
" 'score': 7\n",
"}\n",
"\n",
"step=0\n",
"\n",
"train_costs=[],[]\n",
"test_costs=[],[]\n",
"\n",
"def event_handler(event):\n",
" global step\n",
" global train_costs\n",
" global test_costs\n",
" if isinstance(event, paddle.event.EndIteration):\n",
" need_plot = False\n",
" if step % 10 == 0: # every 10 batches, record a train cost\n",
" train_costs[0].append(step)\n",
" train_costs[1].append(event.cost)\n",
" \n",
" if step % 1000 == 0: # every 1000 batches, record a test cost\n",
" result = trainer.test(reader=paddle.batch(\n",
" paddle.dataset.movielens.test(), batch_size=256))\n",
" test_costs[0].append(step)\n",
" test_costs[1].append(result.cost)\n",
" \n",
" if step % 100 == 0: # every 100 batches, update cost plot\n",
" plt.plot(*train_costs)\n",
" plt.plot(*test_costs)\n",
" plt.legend(['Train Cost', 'Test Cost'], loc='upper left')\n",
" display.clear_output(wait=True)\n",
" display.display(plt.gcf())\n",
" plt.gcf().clear()\n",
" step += 1\n",
"\n",
"trainer.train(\n",
" reader=paddle.batch(\n",
" paddle.reader.shuffle(\n",
" paddle.dataset.movielens.train(), buf_size=8192),\n",
" batch_size=256),\n",
" event_handler=event_handler,\n",
" feeding=feeding,\n",
" num_passes=2)"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": true,
"editable": true
},
"source": [
"## 应用模型\n",
"\n",
"在训练了几轮以后，您可以对模型进行推断。我们可以使用任意一个用户ID和电影ID，来预测该用户对该电影的评分。示例程序为:"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [
},
{
"cell_type": "code",
"metadata": {
"editable": true
},
"source": [
"train_set_creator = paddle.dataset.movielens.train()\n",
"train_sample = next(train_set_creator())\n",
"uid = train_sample[0]\n",
"mov_id = train_sample[len(user_info[uid].value())]\n",
"print \"User %s rates Movie %s with Score %s\"%(user_info[uid], movie_info[mov_id], train_sample[-1])\n"
],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n"
]
}
],
"execution_count": 1
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
" User \u003cUserInfo id(1), gender(F), age(1), job(10)\u003e rates Movie \u003cMovieInfo id(1193), title(One Flew Over the Cuckoo's Nest ), categories(['Drama'])\u003e with Score [5.0]\n",
"\n",
"\n",
"即用户1对电影1193的评价为5分。\n",
"\n",
"## 模型配置说明\n",
"\n",
"下面我们开始根据输入数据的形式配置模型。\n",
"\n",
"\n"
]
},
{
"cell_type": "code",
"metadata": {
"editable": true
},
"source": [
"uid = paddle.layer.data(\n",
" name='user_id',\n",
" type=paddle.data_type.integer_value(\n",
" paddle.dataset.movielens.max_user_id() + 1))\n",
"usr_emb = paddle.layer.embedding(input=uid, size=32)\n",
"\n",
"usr_gender_id = paddle.layer.data(\n",
" name='gender_id', type=paddle.data_type.integer_value(2))\n",
"usr_gender_emb = paddle.layer.embedding(input=usr_gender_id, size=16)\n",
"\n",
"usr_age_id = paddle.layer.data(\n",
" name='age_id',\n",
" type=paddle.data_type.integer_value(\n",
" len(paddle.dataset.movielens.age_table)))\n",
"usr_age_emb = paddle.layer.embedding(input=usr_age_id, size=16)\n",
"\n",
"usr_job_id = paddle.layer.data(\n",
" name='job_id',\n",
" type=paddle.data_type.integer_value(paddle.dataset.movielens.max_job_id(\n",
" ) + 1))\n",
"usr_job_emb = paddle.layer.embedding(input=usr_job_id, size=16)\n"
],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n"
]
}
],
"execution_count": 1
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"如上述代码所示，对于每个用户，我们输入4维特征。其中包括`user_id`,`gender_id`,`age_id`,`job_id`。这几维特征均是简单的整数值。为了后续神经网络处理这些特征方便，我们借鉴NLP中的语言模型，将这几维离散的整数值，变换成embedding取出。分别形成`usr_emb`, `usr_gender_emb`, `usr_age_emb`, `usr_job_emb`。\n",
"\n",
"\n"
]
},
{
"cell_type": "code",
"metadata": {
"editable": true
},
"source": [
"usr_combined_features = paddle.layer.fc(\n",
" input=[usr_emb, usr_gender_emb, usr_age_emb, usr_job_emb],\n",
" size=200,\n",
" act=paddle.activation.Tanh())\n"
],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n"
]
}
],
"execution_count": 1
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"然后，我们对于所有的用户特征，均输入到一个全连接层(fc)中。将所有特征融合为一个200维度的特征。\n",
"\n",
"进而，我们对每一个电影特征做类似的变换，网络配置为:\n",
"\n",
"\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"[INFO 2017-03-06 17:17:08,132 networks.py:1472] The input order is [user_id, gender_id, age_id, job_id, movie_id, category_id, movie_title]\n",
"[INFO 2017-03-06 17:17:08,134 networks.py:1478] The output order is [__cos_sim_0__]\n"
]
"cell_type": "code",
"metadata": {
"editable": true
},
"source": [
"mov_id = paddle.layer.data(\n",
" name='movie_id',\n",
" type=paddle.data_type.integer_value(\n",
" paddle.dataset.movielens.max_movie_id() + 1))\n",
"mov_emb = paddle.layer.embedding(input=mov_id, size=32)\n",
"\n",
"mov_categories = paddle.layer.data(\n",
" name='category_id',\n",
" type=paddle.data_type.sparse_binary_vector(\n",
" len(paddle.dataset.movielens.movie_categories())))\n",
"\n",
"mov_categories_hidden = paddle.layer.fc(input=mov_categories, size=32)\n",
"\n",
"\n",
"movie_title_dict = paddle.dataset.movielens.get_movie_title_dict()\n",
"mov_title_id = paddle.layer.data(\n",
" name='movie_title',\n",
" type=paddle.data_type.integer_value_sequence(len(movie_title_dict)))\n",
"mov_title_emb = paddle.layer.embedding(input=mov_title_id, size=32)\n",
"mov_title_conv = paddle.networks.sequence_conv_pool(\n",
" input=mov_title_emb, hidden_size=32, context_len=3)\n",
"\n",
"mov_combined_features = paddle.layer.fc(\n",
" input=[mov_emb, mov_categories_hidden, mov_title_conv],\n",
" size=200,\n",
" act=paddle.activation.Tanh())\n"
],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n"
]
}
],
"execution_count": 1
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"电影ID和电影类型分别映射到其对应的特征隐层。对于电影标题名称(title)，一个ID序列表示的词语序列，在输入卷积层后，将得到每个时间窗口的特征（序列特征），然后通过在时间维度降采样得到固定维度的特征，整个过程在text_conv_pool实现。\n",
"\n",
"最后再将电影的特征融合进`mov_combined_features`中。\n",
"\n",
"\n"
]
},
{
"cell_type": "code",
"metadata": {
"editable": true
},
"source": [
"inference = paddle.layer.cos_sim(a=usr_combined_features, b=mov_combined_features, size=1, scale=5)\n"
],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n"
]
}
],
"execution_count": 1
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"进而，我们使用余弦相似度计算用户特征与电影特征的相似性。并将这个相似性拟合(回归)到用户评分上。\n",
"\n",
"\n"
]
},
{
"cell_type": "code",
"metadata": {
"editable": true
},
"source": [
"cost = paddle.layer.regression_cost(\n",
" input=inference,\n",
" label=paddle.layer.data(\n",
" name='score', type=paddle.data_type.dense_vector(1)))\n"
],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n"
]
}
],
"execution_count": 1
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"至此，我们的优化目标就是这个网络配置中的`cost`了。\n",
"\n",
"## 训练模型\n",
"\n",
"### 定义参数\n",
"神经网络的模型，我们可以简单的理解为网络拓朴结构+参数。之前一节，我们定义出了优化目标`cost`。这个`cost`即为网络模型的拓扑结构。我们开始训练模型，需要先定义出参数。定义方法为:\n",
"\n",
"\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"[Predict] User 234 Rating Movie 345 With Score 4.16\n"
]
"cell_type": "code",
"metadata": {
"editable": true
},
"source": [
"parameters = paddle.parameters.create(cost)\n"
],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n"
]
}
],
"execution_count": 1
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
" [INFO 2017-03-06 17:12:13,284 networks.py:1472] The input order is [user_id, gender_id, age_id, job_id, movie_id, category_id, movie_title, score]\n",
" [INFO 2017-03-06 17:12:13,287 networks.py:1478] The output order is [__regression_cost_0__]\n",
"\n",
"\n",
"`parameters`是模型的所有参数集合。他是一个python的dict。我们可以查看到这个网络中的所有参数名称。因为之前定义模型的时候，我们没有指定参数名称，这里参数名称是自动生成的。当然，我们也可以指定每一个参数名称，方便日后维护。\n",
"\n",
"\n"
]
},
{
"cell_type": "code",
"metadata": {
"editable": true
},
"source": [
"print parameters.keys()\n"
],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n"
]
}
],
"execution_count": 1
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
" [u'___fc_layer_2__.wbias', u'___fc_layer_2__.w2', u'___embedding_layer_3__.w0', u'___embedding_layer_5__.w0', u'___embedding_layer_2__.w0', u'___embedding_layer_1__.w0', u'___fc_layer_1__.wbias', u'___fc_layer_0__.wbias', u'___fc_layer_1__.w0', u'___fc_layer_0__.w2', u'___fc_layer_0__.w3', u'___fc_layer_0__.w0', u'___fc_layer_0__.w1', u'___fc_layer_2__.w1', u'___fc_layer_2__.w0', u'___embedding_layer_4__.w0', u'___sequence_conv_pool_0___conv_fc.w0', u'___embedding_layer_0__.w0', u'___sequence_conv_pool_0___conv_fc.wbias']\n",
"\n",
"\n",
"### 构造训练(trainer)\n",
"\n",
"下面，我们根据网络拓扑结构和模型参数来构造出一个本地训练(trainer)。在构造本地训练的时候，我们还需要指定这个训练的优化方法。这里我们使用Adam来作为优化算法。\n",
"\n",
"\n"
]
},
{
"cell_type": "code",
"metadata": {
"editable": true
},
"source": [
"trainer = paddle.trainer.SGD(cost=cost, parameters=parameters,\n",
" update_equation=paddle.optimizer.Adam(learning_rate=1e-4))\n"
],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n"
]
}
],
"execution_count": 1
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
" [INFO 2017-03-06 17:12:13,378 networks.py:1472] The input order is [user_id, gender_id, age_id, job_id, movie_id, category_id, movie_title, score]\n",
" [INFO 2017-03-06 17:12:13,379 networks.py:1478] The output order is [__regression_cost_0__]\n",
"\n",
"\n",
"### 训练\n",
"\n",
"下面我们开始训练过程。\n",
"\n",
"我们直接使用Paddle提供的数据集读取程序。`paddle.dataset.movielens.train()`和`paddle.dataset.movielens.test()`分别做训练和预测数据集。并且通过`reader_dict`来指定每一个数据和data_layer的对应关系。\n",
"\n",
"例如，这里的reader_dict表示的是，对于数据层 `user_id`，使用了reader中每一条数据的第0个元素。`gender_id`数据层使用了第1个元素。以此类推。\n",
"\n",
"训练过程是完全自动的。我们可以使用event_handler来观察训练过程，或进行测试等。这里我们在event_handler里面绘制了训练误差曲线和测试误差曲线。并且保存了模型。\n",
"\n",
"\n"
]
},
{
"cell_type": "code",
"metadata": {
"editable": true
},
"source": [
"%matplotlib inline\n",
"\n",
"import matplotlib.pyplot as plt\n",
"from IPython import display\n",
"import cPickle\n",
"\n",
"feeding = {\n",
" 'user_id': 0,\n",
" 'gender_id': 1,\n",
" 'age_id': 2,\n",
" 'job_id': 3,\n",
" 'movie_id': 4,\n",
" 'category_id': 5,\n",
" 'movie_title': 6,\n",
" 'score': 7\n",
"}\n",
"\n",
"step=0\n",
"\n",
"train_costs=[],[]\n",
"test_costs=[],[]\n",
"\n",
"def event_handler(event):\n",
" global step\n",
" global train_costs\n",
" global test_costs\n",
" if isinstance(event, paddle.event.EndIteration):\n",
" need_plot = False\n",
" if step % 10 == 0: # every 10 batches, record a train cost\n",
" train_costs[0].append(step)\n",
" train_costs[1].append(event.cost)\n",
"\n",
" if step % 1000 == 0: # every 1000 batches, record a test cost\n",
" result = trainer.test(reader=paddle.batch(\n",
" paddle.dataset.movielens.test(), batch_size=256))\n",
" test_costs[0].append(step)\n",
" test_costs[1].append(result.cost)\n",
"\n",
" if step % 100 == 0: # every 100 batches, update cost plot\n",
" plt.plot(*train_costs)\n",
" plt.plot(*test_costs)\n",
" plt.legend(['Train Cost', 'Test Cost'], loc='upper left')\n",
" display.clear_output(wait=True)\n",
" display.display(plt.gcf())\n",
" plt.gcf().clear()\n",
" step += 1\n",
"\n",
"trainer.train(\n",
" reader=paddle.batch(\n",
" paddle.reader.shuffle(\n",
" paddle.dataset.movielens.train(), buf_size=8192),\n",
" batch_size=256),\n",
" event_handler=event_handler,\n",
" feeding=feeding,\n",
" num_passes=2)\n"
],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n"
]
}
],
"execution_count": 1
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"\n",
"![png](./image/output_32_0.png)\n",
"\n",
"## 应用模型\n",
"\n",
"在训练了几轮以后，您可以对模型进行推断。我们可以使用任意一个用户ID和电影ID，来预测该用户对该电影的评分。示例程序为:\n",
"\n",
"\n"
]
},
{
"cell_type": "code",
"metadata": {
"editable": true
},
"source": [
"import copy\n",
"user_id = 234\n",
"movie_id = 345\n",
"\n",
"user = user_info[user_id]\n",
"movie = movie_info[movie_id]\n",
"\n",
"feature = user.value() + movie.value()\n",
"\n",
"infer_dict = copy.copy(feeding)\n",
"del infer_dict['score']\n",
"\n",
"prediction = paddle.infer(output=inference, parameters=parameters, input=[feature], feeding=infer_dict)\n",
"score = (prediction[0][0] + 5.0) / 2\n",
"print \"[Predict] User %d Rating Movie %d With Score %.2f\"%(user_id, movie_id, score)\n"
],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n"
]
}
],
"execution_count": 1
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
" [INFO 2017-03-06 17:17:08,132 networks.py:1472] The input order is [user_id, gender_id, age_id, job_id, movie_id, category_id, movie_title]\n",
" [INFO 2017-03-06 17:17:08,134 networks.py:1478] The output order is [__cos_sim_0__]\n",
"\n",
"\n",
" [Predict] User 234 Rating Movie 345 With Score 4.16\n",
"\n",
"\n",
"## 总结\n",
"\n",
"本章介绍了传统的推荐系统方法和YouTube的深度神经网络推荐系统，并以电影推荐为例，使用PaddlePaddle训练了一个个性化推荐神经网络模型。推荐系统几乎涵盖了电商系统、社交网络、广告推荐、搜索引擎等领域的方方面面，而在图像处理、自然语言处理等领域已经发挥重要作用的深度学习技术，也将会在推荐系统领域大放异彩。\n",
"\n",
"## 参考文献\n",
"\n",
"1. [Peter Brusilovsky](https://en.wikipedia.org/wiki/Peter_Brusilovsky) (2007). *The Adaptive Web*. p. 325.\n",
"2. Robin Burke , [Hybrid Web Recommender Systems](http://www.dcs.warwick.ac.uk/~acristea/courses/CS411/2010/Book%20-%20The%20Adaptive%20Web/HybridWebRecommenderSystems.pdf), pp. 377-408, The Adaptive Web, Peter Brusilovsky, Alfred Kobsa, Wolfgang Nejdl (Ed.), Lecture Notes in Computer Science, Springer-Verlag, Berlin, Germany, Lecture Notes in Computer Science, Vol. 4321, May 2007, 978-3-540-72078-2.\n",
"3. P. Resnick, N. Iacovou, etc. “[GroupLens: An Open Architecture for Collaborative Filtering of Netnews](http://ccs.mit.edu/papers/CCSWP165.html)”, Proceedings of ACM Conference on Computer Supported Cooperative Work, CSCW 1994. pp.175-186.\n",
"4. Sarwar, Badrul, et al. \"[Item-based collaborative filtering recommendation algorithms.](http://files.grouplens.org/papers/www10_sarwar.pdf)\" *Proceedings of the 10th international conference on World Wide Web*. ACM, 2001.\n",
"5. Kautz, Henry, Bart Selman, and Mehul Shah. \"[Referral Web: combining social networks and collaborative filtering.](http://www.cs.cornell.edu/selman/papers/pdf/97.cacm.refweb.pdf)\" Communications of the ACM 40.3 (1997): 63-65. APA\n",
"6. Yuan, Jianbo, et al. [\"Solving Cold-Start Problem in Large-scale Recommendation Engines: A Deep Learning Approach.\"](https://arxiv.org/pdf/1611.05480v1.pdf) *arXiv preprint arXiv:1611.05480* (2016).\n",
"7. Covington P, Adams J, Sargin E. [Deep neural networks for youtube recommendations](https://static.googleusercontent.com/media/research.google.com/zh-CN//pubs/archive/45530.pdf)[C]//Proceedings of the 10th ACM Conference on Recommender Systems. ACM, 2016: 191-198.\n",
"\n",
"\u003cbr/\u003e\n",
"\u003ca rel=\"license\" href=\"http://creativecommons.org/licenses/by-nc-sa/4.0/\"\u003e\u003cimg alt=\"知识共享许可协议\" style=\"border-width:0\" src=\"https://i.creativecommons.org/l/by-nc-sa/4.0/88x31.png\" /\u003e\u003c/a\u003e\u003cbr /\u003e\u003cspan xmlns:dct=\"http://purl.org/dc/terms/\" href=\"http://purl.org/dc/dcmitype/Text\" property=\"dct:title\" rel=\"dct:type\"\u003e本教程\u003c/span\u003e 由 \u003ca xmlns:cc=\"http://creativecommons.org/ns#\" href=\"http://book.paddlepaddle.org\" property=\"cc:attributionName\" rel=\"cc:attributionURL\"\u003ePaddlePaddle\u003c/a\u003e 创作，采用 \u003ca rel=\"license\" href=\"http://creativecommons.org/licenses/by-nc-sa/4.0/\"\u003e知识共享 署名-非商业性使用-相同方式共享 4.0 国际 许可协议\u003c/a\u003e进行许可。\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.0"
}
],
"source": [
"import copy\n",
"user_id = 234\n",
"movie_id = 345\n",
"\n",
"user = user_info[user_id]\n",
"movie = movie_info[movie_id]\n",
"\n",
"feature = user.value() + movie.value()\n",
"\n",
"infer_dict = copy.copy(feeding)\n",
"del infer_dict['score']\n",
"\n",
"prediction = paddle.infer(output=inference, parameters=parameters, input=[feature], feeding=infer_dict)\n",
"score = (prediction[0][0] + 5.0) / 2\n",
"print \"[Predict] User %d Rating Movie %d With Score %.2f\"%(user_id, movie_id, score)"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": true,
"editable": true
},
"source": [
"## 总结\n",
"\n",
"本章介绍了传统的推荐系统方法和YouTube的深度神经网络推荐系统，并以电影推荐为例，使用PaddlePaddle训练了一个个性化推荐神经网络模型。推荐系统几乎涵盖了电商系统、社交网络、广告推荐、搜索引擎等领域的方方面面，而在图像处理、自然语言处理等领域已经发挥重要作用的深度学习技术，也将会在推荐系统领域大放异彩。\n",
"\n",
"## 参考文献\n",
"\n",
"1. [Peter Brusilovsky](https://en.wikipedia.org/wiki/Peter_Brusilovsky) (2007). *The Adaptive Web*. p. 325.\n",
"2. Robin Burke , [Hybrid Web Recommender Systems](http://www.dcs.warwick.ac.uk/~acristea/courses/CS411/2010/Book%20-%20The%20Adaptive%20Web/HybridWebRecommenderSystems.pdf), pp. 377-408, The Adaptive Web, Peter Brusilovsky, Alfred Kobsa, Wolfgang Nejdl (Ed.), Lecture Notes in Computer Science, Springer-Verlag, Berlin, Germany, Lecture Notes in Computer Science, Vol. 4321, May 2007, 978-3-540-72078-2.\n",
"3. P. Resnick, N. Iacovou, etc. “[GroupLens: An Open Architecture for Collaborative Filtering of Netnews](http://ccs.mit.edu/papers/CCSWP165.html)”, Proceedings of ACM Conference on Computer Supported Cooperative Work, CSCW 1994. pp.175-186.\n",
"4. Sarwar, Badrul, et al. \"[Item-based collaborative filtering recommendation algorithms.](http://files.grouplens.org/papers/www10_sarwar.pdf)\" *Proceedings of the 10th international conference on World Wide Web*. ACM, 2001.\n",
"5. Kautz, Henry, Bart Selman, and Mehul Shah. \"[Referral Web: combining social networks and collaborative filtering.](http://www.cs.cornell.edu/selman/papers/pdf/97.cacm.refweb.pdf)\" Communications of the ACM 40.3 (1997): 63-65. APA\n",
"6. Yuan, Jianbo, et al. [\"Solving Cold-Start Problem in Large-scale Recommendation Engines: A Deep Learning Approach.\"](https://arxiv.org/pdf/1611.05480v1.pdf) *arXiv preprint arXiv:1611.05480* (2016).\n",
"7. Covington P, Adams J, Sargin E. [Deep neural networks for youtube recommendations](https://static.googleusercontent.com/media/research.google.com/zh-CN//pubs/archive/45530.pdf)[C]//Proceedings of the 10th ACM Conference on Recommender Systems. ACM, 2016: 191-198.\n",
"\n",
"<br/>\n",
"<a rel=\"license\" href=\"http://creativecommons.org/licenses/by-nc-sa/4.0/\"><img alt=\"知识共享许可协议\" style=\"border-width:0\" src=\"https://i.creativecommons.org/l/by-nc-sa/4.0/88x31.png\" /></a><br /><span xmlns:dct=\"http://purl.org/dc/terms/\" href=\"http://purl.org/dc/dcmitype/Text\" property=\"dct:title\" rel=\"dct:type\">本教程</span> 由 <a xmlns:cc=\"http://creativecommons.org/ns#\" href=\"http://book.paddlepaddle.org\" property=\"cc:attributionName\" rel=\"cc:attributionURL\">PaddlePaddle</a> 创作，采用 <a rel=\"license\" href=\"http://creativecommons.org/licenses/by-nc-sa/4.0/\">知识共享 署名-非商业性使用-相同方式共享 4.0 国际 许可协议</a>进行许可。\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 2",
"language": "python",
"name": "python2"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 2
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython2",
"version": "2.7.13"
}
},
"nbformat": 4,
"nbformat_minor": 0
"nbformat": 4,
"nbformat_minor": 0
}
tools/convert-markdown-into-ipynb-and-test.sh 0 → 100755
浏览文件 @ 2f6da124
#!/bin/sh
command -v go >/dev/null 2>&1
if [ $? -ne 0 ]; then
echo >&2 "Please install go https://golang.org/doc/install#install"
exit 1
fi
GOPATH=/tmp/go go get -u github.com/wangkuiyi/ipynb/markdown-to-ipynb
cur_path=$(dirname $(readlink -f $0))
cd $cur_path/../
#convert md to ipynb
for file in */{README,README\.en}.md ; do
/tmp/go/bin/markdown-to-ipynb < $file > ${file%.*}".ipynb"
if [ $? -ne 0 ]; then
echo >&2 "markdown-to-ipynb $file error"
exit 1
fi
done
if [[ -z $TEST_EMBEDDED_PYTHON_SCRIPTS ]]; then
exit 0
fi
#exec ipynb's py file
for file in */{README,README\.en}.ipynb ; do
pushd $PWD > /dev/null
cd $(dirname $file) > /dev/null
echo "begin test $file"
jupyter nbconvert --to python $(basename $file) --stdout | python
popd > /dev/null
#break
done
Markdown is supported
0% 或 .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑！
想要评论请 注册 或