{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "V0CqD2QfXd9R"
   },
   "source": [
    "# 线性回归\n",
    "NOTE: 本示例教程是基于2.0beta版本开发"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "6lPmRFntXYIp"
   },
   "source": [
    "## 简要介绍\n",
    "经典的线性回归模型主要用来预测一些存在着线性关系的数据集。回归模型可以理解为：存在一个点集，用一条曲线去拟合它分布的过程。如果拟合曲线是一条直线，则称为线性回归。如果是一条二次曲线，则被称为二次回归。线性回归是回归模型中最简单的一种。 \n",
    "本示例简要介绍如何用飞桨开源框架，实现波士顿房价预测。其思路是，假设uci-housing数据集中的房子属性和房价之间的关系可以被属性间的线性组合描述。在模型训练阶段，让假设的预测结果和真实值之间的误差越来越小。在模型预测阶段，预测器会读取训练好的模型，对从未遇见过的房子属性进行房价预测。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "OEOMtGXCZaRR"
   },
   "source": [
    "## 数据集介绍\n",
    "本示例采用uci-housing数据集，这是经典线性回归的数据集。数据集共7084条数据，可以拆分成506行,每行14列。前13列用来描述房屋的各种信息，最后一列为该类房屋价格中位数。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "前13列用来描述房屋的各种信息\n",
    "\n",
    "![avatar](https://ai-studio-static-online.cdn.bcebos.com/c19602ce74284e3b9a50422f8dc37c0c1c79cf5cd8424994b6a6b073dcb7c057)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 训练方式一\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 环境设置"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2.0.0-beta0'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import paddle\n",
    "import numpy as np\n",
    "import os\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "\n",
    "paddle.disable_static()\n",
    "paddle.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 数据处理"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "#下载数据\n",
    "#!wget https://archive.ics.uci.edu/ml/machine-learning-databases/housing/housing.data -O housing.data "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 从文件导入数据\n",
    "datafile = './housing.data'\n",
    "housing_data = np.fromfile(datafile, sep=' ')\n",
    "\n",
    "feature_names = ['CRIM', 'ZN', 'INDUS', 'CHAS', 'NOX', 'RM', 'AGE','DIS', 'RAD', 'TAX', 'PTRATIO', 'B', 'LSTAT', 'MEDV']\n",
    "feature_num = len(feature_names)\n",
    "# 将原始数据进行Reshape，变成[N, 14]这样的形状\n",
    "housing_data = housing_data.reshape([housing_data.shape[0] // feature_num, feature_num])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 137316 characters]\n",
      "text/plain": [
       "<Figure size 2520x180 with 14 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 画图看特征间的关系,主要是变量两两之间的关系（线性或非线性，有无明显较为相关关系）\n",
    "features_np = np.array([x[:13] for x in housing_data], np.float32)\n",
    "labels_np = np.array([x[-1] for x in housing_data], np.float32)\n",
    "data_np = np.c_[features_np, labels_np]\n",
    "df = pd.DataFrame(data_np, columns=feature_names)\n",
    "matplotlib.use('TkAgg')\n",
    "%matplotlib inline\n",
    "sns.pairplot(df.dropna(), y_vars=feature_names[-1], x_vars=feature_names[:])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 12876 characters]\n",
      "text/plain": [
       "<Figure size 1080x72 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 相关性分析\n",
    "fig, ax = plt.subplots(figsize=(15, 1)) \n",
    "corr_data = df.corr().iloc[-1]\n",
    "corr_data = np.asarray(corr_data).reshape(1, 14)\n",
    "ax = sns.heatmap(corr_data, cbar=True, annot=True)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "IUhqen8LWAYM"
   },
   "source": [
    "***数据归一化处理***<br>\n",
    "下图为大家展示各属性的取值范围分布："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x1a3e2b4e50>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 12632 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.boxplot(data=df.iloc[:, 0:13])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "从上图看出，我们各属性的数值范围差异太大，甚至不能够在一个画布上充分的展示各属性具体的最大、最小值以及异常值等。下面我们进行归一化。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "做归一化（或 Feature scaling）至少有以下2个理由：\n",
    "\n",
    "*   过大或过小的数值范围会导致计算时的浮点上溢或下溢。\n",
    "*   不同的数值范围会导致不同属性对模型的重要性不同（至少在训练的初始阶段如此），而这个隐含的假设常常是不合理的。这会对优化的过程造成困难，使训练时间大大的加长.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "features_max = housing_data.max(axis=0)\n",
    "features_min = housing_data.min(axis=0)\n",
    "features_avg = housing_data.sum(axis=0) / housing_data.shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "BATCH_SIZE = 20\n",
    "def feature_norm(input):\n",
    "    f_size = input.shape\n",
    "    output_features = np.zeros(f_size, np.float32)\n",
    "    for batch_id in range(f_size[0]):\n",
    "        for index in range(13):\n",
    "            output_features[batch_id][index] = (input[batch_id][index] - features_avg[index]) / (features_max[index] - features_min[index])\n",
    "    return output_features "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "#只对属性进行归一化\n",
    "housing_features = feature_norm(housing_data[:, :13])\n",
    "# print(feature_trian.shape)\n",
    "housing_data = np.c_[housing_features, housing_data[:, -1]].astype(np.float32)\n",
    "# print(training_data[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x1a3e4cd4d0>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 16228 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#归一化后的train_data,我们看下各属性的情况\n",
    "features_np = np.array([x[:13] for x in housing_data],np.float32)\n",
    "labels_np = np.array([x[-1] for x in housing_data],np.float32)\n",
    "data_np = np.c_[features_np, labels_np]\n",
    "df = pd.DataFrame(data_np, columns=feature_names)\n",
    "sns.boxplot(data=df.iloc[:, 0:13])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "#将训练数据集和测试数据集按照8:2的比例分开\n",
    "ratio = 0.8\n",
    "offset = int(housing_data.shape[0] * ratio)\n",
    "train_data = housing_data[:offset]\n",
    "test_data = housing_data[offset:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "JkEt541Cl0s8"
   },
   "source": [
    "### 模型配置\n",
    "线性回归就是一个从输入到输出的简单的全连接层。\n",
    "\n",
    "对于波士顿房价数据集，假设属性和房价之间的关系可以被属性间的线性组合描述。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "JW2IHEVbbCI3"
   },
   "outputs": [],
   "source": [
    "class Regressor(paddle.nn.Layer):\n",
    "    def __init__(self):\n",
    "        super(Regressor, self).__init__()\n",
    "        self.fc = paddle.nn.Linear(13, 1,)\n",
    "\n",
    "    def forward(self, inputs):\n",
    "        pred = self.fc(inputs)\n",
    "        return pred"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "定义绘制训练过程的损失值变化趋势的方法draw_train_process"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_nums = []\n",
    "train_costs = []\n",
    "\n",
    "def draw_train_process(iters, train_costs):\n",
    "    plt.title(\"training cost\", fontsize=24)\n",
    "    plt.xlabel(\"iter\", fontsize=14)\n",
    "    plt.ylabel(\"cost\", fontsize=14)\n",
    "    plt.plot(iters, train_costs, color='red', label='training cost')\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "oxD989B_cBjF"
   },
   "source": [
    "### 模型训练\n",
    "下面为大家展示模型训练的代码。\n",
    "这里用到的是线性回归模型最常用的损失函数--均方误差（MSE），用来衡量模型预测的房价和真实房价的差异。\n",
    "对损失函数进行优化所采用的方法是梯度下降法"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 562
    },
    "colab_type": "code",
    "id": "m2as0xbvc6Ec",
    "outputId": "18c62056-d21b-4b7b-b622-4b4b28f4ff93"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "start training ... \n",
      "Pass:0,Cost:740.21814\n",
      "Pass:50,Cost:36.40338\n",
      "Pass:100,Cost:86.01823\n",
      "Pass:150,Cost:50.86654\n",
      "Pass:200,Cost:31.14208\n",
      "Pass:250,Cost:20.54596\n",
      "Pass:300,Cost:22.30817\n",
      "Pass:350,Cost:24.18756\n",
      "Pass:400,Cost:22.22965\n",
      "Pass:450,Cost:39.25978\n"
     ]
    }
   ],
   "source": [
    "import paddle.nn.functional as F \n",
    "y_preds = []\n",
    "labels_list = []\n",
    "\n",
    "def train(model):\n",
    "    print('start training ... ')\n",
    "    # 开启模型训练模式\n",
    "    model.train()\n",
    "    EPOCH_NUM = 500\n",
    "    train_num = 0\n",
    "    optimizer = paddle.optimizer.SGD(learning_rate=0.001, parameters=model.parameters())\n",
    "    for epoch_id in range(EPOCH_NUM):\n",
    "        # 在每轮迭代开始之前，将训练数据的顺序随机的打乱\n",
    "        np.random.shuffle(train_data)\n",
    "        # 将训练数据进行拆分，每个batch包含20条数据\n",
    "        mini_batches = [train_data[k:k+BATCH_SIZE] for k in range(0, len(train_data), BATCH_SIZE)]\n",
    "        for batch_id, data in enumerate(mini_batches):\n",
    "            features_np = np.array(data[:, :13], np.float32)\n",
    "            labels_np = np.array(data[:, -1:], np.float32)\n",
    "            features = paddle.to_tensor(features_np)\n",
    "            labels = paddle.to_tensor(labels_np)\n",
    "            #前向计算\n",
    "            y_pred = model(features)\n",
    "            cost = F.mse_loss(y_pred, label=labels)\n",
    "            train_cost = cost.numpy()[0]\n",
    "            #反向传播\n",
    "            cost.backward()\n",
    "            #最小化loss，更新参数\n",
    "            optimizer.step()\n",
    "            # 清除梯度\n",
    "            optimizer.clear_grad()\n",
    "            \n",
    "            if batch_id%30 == 0 and epoch_id%50 == 0:\n",
    "                print(\"Pass:%d,Cost:%0.5f\"%(epoch_id, train_cost))\n",
    "\n",
    "            train_num = train_num + BATCH_SIZE\n",
    "            train_nums.append(train_num)\n",
    "            train_costs.append(train_cost)\n",
    "        \n",
    "model = Regressor()\n",
    "train(model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 15968 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "matplotlib.use('TkAgg')\n",
    "%matplotlib inline\n",
    "draw_train_process(train_nums, train_costs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以从上图看出，随着训练轮次的增加，损失在呈降低趋势。但由于每次仅基于少量样本更新参数和计算损失，所以损失下降曲线会出现震荡。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "YC73FnkakWbY"
   },
   "source": [
    "### 模型预测\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No.0: infer result is 12.15,ground truth is 8.50\n",
      "No.10: infer result is 5.21,ground truth is 7.00\n",
      "No.20: infer result is 14.32,ground truth is 11.70\n",
      "No.30: infer result is 16.11,ground truth is 11.70\n",
      "No.40: infer result is 13.42,ground truth is 10.80\n",
      "No.50: infer result is 15.50,ground truth is 14.90\n",
      "No.60: infer result is 18.81,ground truth is 21.40\n",
      "No.70: infer result is 15.42,ground truth is 13.80\n",
      "No.80: infer result is 18.16,ground truth is 20.60\n",
      "No.90: infer result is 21.48,ground truth is 24.50\n",
      "Mean loss is: [12.195988]\n"
     ]
    }
   ],
   "source": [
    "#获取预测数据\n",
    "INFER_BATCH_SIZE = 100\n",
    "\n",
    "infer_features_np = np.array([data[:13] for data in test_data]).astype(\"float32\")\n",
    "infer_labels_np = np.array([data[-1] for data in test_data]).astype(\"float32\")\n",
    "\n",
    "infer_features = paddle.to_tensor(infer_features_np)\n",
    "infer_labels = paddle.to_tensor(infer_labels_np)\n",
    "fetch_list = model(infer_features)\n",
    "\n",
    "sum_cost = 0\n",
    "for i in range(INFER_BATCH_SIZE):\n",
    "    infer_result = fetch_list[i][0]\n",
    "    ground_truth = infer_labels[i]\n",
    "    if i % 10 == 0:\n",
    "        print(\"No.%d: infer result is %.2f,ground truth is %.2f\" % (i, infer_result, ground_truth))\n",
    "    cost = paddle.pow(infer_result - ground_truth, 2)\n",
    "    sum_cost += cost\n",
    "mean_loss = sum_cost / INFER_BATCH_SIZE\n",
    "print(\"Mean loss is:\", mean_loss.numpy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_pred_ground(pred, ground):\n",
    "    plt.figure()   \n",
    "    plt.title(\"Predication v.s. Ground truth\", fontsize=24)\n",
    "    plt.xlabel(\"ground truth price(unit:$1000)\", fontsize=14)\n",
    "    plt.ylabel(\"predict price\", fontsize=14)\n",
    "    plt.scatter(ground, pred, alpha=0.5)  #  scatter:散点图,alpha:\"透明度\"\n",
    "    plt.plot(ground, ground, c='red')\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 23776 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_pred_ground(fetch_list, infer_labels_np)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上图可以看出，我们训练出来的模型的预测结果与真实结果是较为接近的。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 训练方式二\n",
    "我们也可以用我们的高层API来做线性回归训练，高层API相较于底层API更加的简洁方便。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "step 20/51 - loss: 520.8663 - 1ms/step\n",
      "step 40/51 - loss: 611.7135 - 1ms/step\n",
      "step 51/51 - loss: 620.0662 - 1ms/step\n",
      "Eval begin...\n",
      "step 13/13 - loss: 389.7871 - 1ms/step\n",
      "Eval samples: 102\n",
      "Epoch 2/5\n",
      "step 20/51 - loss: 867.4678 - 3ms/step\n",
      "step 40/51 - loss: 1081.1701 - 2ms/step\n",
      "step 51/51 - loss: 420.8705 - 2ms/step\n",
      "Eval begin...\n",
      "step 13/13 - loss: 387.2432 - 1ms/step\n",
      "Eval samples: 102\n",
      "Epoch 3/5\n",
      "step 20/51 - loss: 810.1555 - 2ms/step\n",
      "step 40/51 - loss: 840.3570 - 2ms/step\n",
      "step 51/51 - loss: 421.0806 - 2ms/step\n",
      "Eval begin...\n",
      "step 13/13 - loss: 384.7417 - 693us/step\n",
      "Eval samples: 102\n",
      "Epoch 4/5\n",
      "step 20/51 - loss: 647.1215 - 1ms/step\n",
      "step 40/51 - loss: 682.9673 - 1ms/step\n",
      "step 51/51 - loss: 422.0570 - 1ms/step\n",
      "Eval begin...\n",
      "step 13/13 - loss: 382.2546 - 591us/step\n",
      "Eval samples: 102\n",
      "Epoch 5/5\n",
      "step 20/51 - loss: 713.3719 - 1ms/step\n",
      "step 40/51 - loss: 567.0962 - 1ms/step\n",
      "step 51/51 - loss: 456.8702 - 1ms/step\n",
      "Eval begin...\n",
      "step 13/13 - loss: 379.7527 - 985us/step\n",
      "Eval samples: 102\n"
     ]
    }
   ],
   "source": [
    "import paddle\n",
    "paddle.disable_static()\n",
    "paddle.set_default_dtype(\"float64\")\n",
    "\n",
    "#step1:用高层API定义数据集，无需进行数据处理等，高层API为您一条龙搞定\n",
    "train_dataset = paddle.text.datasets.UCIHousing(mode='train')\n",
    "eval_dataset = paddle.text.datasets.UCIHousing(mode='test')\n",
    "\n",
    "#step2:定义模型\n",
    "class UCIHousing(paddle.nn.Layer):\n",
    "    def __init__(self):\n",
    "        super(UCIHousing, self).__init__()\n",
    "        self.fc = paddle.nn.Linear(13, 1, None)\n",
    "\n",
    "    def forward(self, input):\n",
    "        pred = self.fc(input)\n",
    "        return pred\n",
    "\n",
    "#step3:训练模型\n",
    "model = paddle.Model(UCIHousing())\n",
    "model.prepare(paddle.optimizer.Adam(parameters=model.parameters()),\n",
    "              paddle.nn.loss.MSELoss())\n",
    "model.fit(train_dataset, eval_dataset, epochs=5, batch_size=8, log_freq=20)"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "线性回归（基于paddle2.0-alpha）",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
