{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 动态图\n",
    "\n",
    "从飞桨开源框架2.0beta版本开始，飞桨默认为用户开启了动态图模式。在这种模式下，每次执行一个运算，可以立即得到结果（而不是事先定义好网络结构，然后再执行）。\n",
    "\n",
    "在动态图模式下，您可以更加方便的组织代码，更容易的调试程序，本示例教程将向你介绍飞桨的动态图的使用。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 设置环境\n",
    "\n",
    "我们将使用飞桨2.0beta版本，并确认已经开启了动态图模式。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.0.0-beta0\n"
     ]
    }
   ],
   "source": [
    "import paddle\n",
    "import paddle.nn.functional as F\n",
    "import numpy as np\n",
    "\n",
    "paddle.disable_static()\n",
    "print(paddle.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 基本用法\n",
    "\n",
    "在动态图模式下，您可以直接运行一个飞桨提供的API，它会立刻返回结果到python。不再需要首先创建一个计算图，然后再给定数据去运行。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 1.5645729  -0.74514765]\n",
      " [-0.01248     0.68240154]\n",
      " [ 0.11316949 -1.6579045 ]\n",
      " [-0.1425675  -1.0153968 ]]\n",
      "[1. 2.]\n",
      "[[2.5645728  1.2548523 ]\n",
      " [0.98752    2.6824017 ]\n",
      " [1.1131694  0.3420955 ]\n",
      " [0.8574325  0.98460317]]\n",
      "[ 0.07427764  1.352323   -3.2026396  -2.173361  ]\n"
     ]
    }
   ],
   "source": [
    "a = paddle.randn([4, 2])\n",
    "b = paddle.arange(1, 3, dtype='float32')\n",
    "\n",
    "print(a.numpy())\n",
    "print(b.numpy())\n",
    "\n",
    "c = a + b\n",
    "print(c.numpy())\n",
    "\n",
    "d = paddle.matmul(a, b)\n",
    "print(d.numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 使用python的控制流\n",
    "\n",
    "动态图模式下，您可以使用python的条件判断和循环，这类控制语句来执行神经网络的计算。（不再需要`cond`, `loop`这类OP)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 +> [5 6 7]\n",
      "1 -> [-3 -3 -3]\n",
      "2 +> [ 5  9 15]\n",
      "3 -> [-3  3 21]\n",
      "4 +> [ 5 21 87]\n",
      "5 -> [ -3  27 237]\n",
      "6 -> [ -3  59 723]\n",
      "7 -> [  -3  123 2181]\n",
      "8 +> [   5  261 6567]\n",
      "9 +> [    5   517 19689]\n"
     ]
    }
   ],
   "source": [
    "a = paddle.to_tensor(np.array([1, 2, 3]))\n",
    "b = paddle.to_tensor(np.array([4, 5, 6]))\n",
    "\n",
    "for i in range(10):\n",
    "    r = paddle.rand([1,])\n",
    "    if r > 0.5:\n",
    "        c = paddle.pow(a, i) + b\n",
    "        print(\"{} +> {}\".format(i, c.numpy()))\n",
    "    else:\n",
    "        c = paddle.pow(a, i) - b\n",
    "        print(\"{} -> {}\".format(i, c.numpy()))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 构建更加灵活的网络：控制流\n",
    "\n",
    "- 使用动态图可以用来创建更加灵活的网络，比如根据控制流选择不同的分支网络，和方便的构建权重共享的网络。接下来我们来看一个具体的例子，在这个例子中，第二个线性变换只有0.5的可能性会运行。\n",
    "- 在sequence to sequence with attention的机器翻译的示例中，你会看到更实际的使用动态图构建RNN类的网络带来的灵活性。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MyModel(paddle.nn.Layer):\n",
    "    def __init__(self, input_size, hidden_size):\n",
    "        super(MyModel, self).__init__()\n",
    "        self.linear1 = paddle.nn.Linear(input_size, hidden_size)\n",
    "        self.linear2 = paddle.nn.Linear(hidden_size, hidden_size)\n",
    "        self.linear3 = paddle.nn.Linear(hidden_size, 1)\n",
    "\n",
    "    def forward(self, inputs):\n",
    "        x = self.linear1(inputs)\n",
    "        x = F.relu(x)\n",
    "\n",
    "        if paddle.rand([1,]) > 0.5: \n",
    "            x = self.linear2(x)\n",
    "            x = F.relu(x)\n",
    "\n",
    "        x = self.linear3(x)\n",
    "        \n",
    "        return x     "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 [1.3384138]\n",
      "200 [0.7855983]\n",
      "400 [0.59084535]\n",
      "600 [0.30849028]\n",
      "800 [0.26992702]\n",
      "1000 [0.03990713]\n",
      "1200 [0.07111286]\n",
      "1400 [0.01177792]\n",
      "1600 [0.03160322]\n",
      "1800 [0.02757282]\n",
      "2000 [0.00916022]\n",
      "2200 [0.00217024]\n",
      "2400 [0.00186833]\n",
      "2600 [0.00101926]\n",
      "2800 [0.0009654]\n"
     ]
    }
   ],
   "source": [
    "total_data, batch_size, input_size, hidden_size = 1000, 64, 128, 256\n",
    "\n",
    "x_data = np.random.randn(total_data, input_size).astype(np.float32)\n",
    "y_data = np.random.randn(total_data, 1).astype(np.float32)\n",
    "\n",
    "model = MyModel(input_size, hidden_size)\n",
    "\n",
    "loss_fn = paddle.nn.MSELoss(reduction='mean')\n",
    "optimizer = paddle.optimizer.SGD(learning_rate=0.01, \n",
    "                                 parameters=model.parameters())\n",
    "\n",
    "for t in range(200 * (total_data // batch_size)):\n",
    "    idx = np.random.choice(total_data, batch_size, replace=False)\n",
    "    x = paddle.to_tensor(x_data[idx,:])\n",
    "    y = paddle.to_tensor(y_data[idx,:])\n",
    "    y_pred = model(x)\n",
    "\n",
    "    loss = loss_fn(y_pred, y)\n",
    "    if t % 200 == 0:\n",
    "        print(t, loss.numpy())\n",
    "\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "    optimizer.clear_grad()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 构建更加灵活的网络：共享权重\n",
    "\n",
    "- 使用动态图还可以更加方便的创建共享权重的网络，下面的示例展示了一个共享了权重的简单的AutoEncoder。\n",
    "- 你也可以参考图像搜索的示例看到共享参数权重的更实际的使用。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step: 0, loss: [0.33474904]\n",
      "step: 1, loss: [0.31669515]\n",
      "step: 2, loss: [0.29729688]\n",
      "step: 3, loss: [0.27288628]\n",
      "step: 4, loss: [0.24694422]\n",
      "step: 5, loss: [0.2203041]\n",
      "step: 6, loss: [0.19171436]\n",
      "step: 7, loss: [0.16213782]\n",
      "step: 8, loss: [0.13443354]\n",
      "step: 9, loss: [0.11170781]\n"
     ]
    }
   ],
   "source": [
    "inputs = paddle.rand((256, 64))\n",
    "\n",
    "linear = paddle.nn.Linear(64, 8, bias_attr=False)\n",
    "loss_fn = paddle.nn.MSELoss()\n",
    "optimizer = paddle.optimizer.Adam(0.01, parameters=linear.parameters())\n",
    "\n",
    "for i in range(10):\n",
    "    hidden = linear(inputs)\n",
    "    # weight from input to hidden is shared with the linear mapping from hidden to output\n",
    "    outputs = paddle.matmul(hidden, linear.weight, transpose_y=True) \n",
    "    loss = loss_fn(outputs, inputs)\n",
    "    loss.backward()\n",
    "    print(\"step: {}, loss: {}\".format(i, loss.numpy()))\n",
    "    optimizer.step()\n",
    "    optimizer.clear_grad()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The end\n",
    "\n",
    "可以看到使用动态图带来了更灵活易用的方式来组网和训练。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
