Merge branch 'graph' into master

codes/ipython/1-basics/graph.ipynb

+{
+  "nbformat": 4,
+  "nbformat_minor": 0,
+  "metadata": {
+    "colab": {
+      "name": "graph.ipynb",
+      "provenance": [],
+      "collapsed_sections": []
+    },
+    "kernelspec": {
+      "name": "python3",
+      "display_name": "Python 3"
+    }
+  },
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "MDevcewD85Im"
+      },
+      "source": [
+        "## Intorduction to TensorFlow graphs.\n",
+        "\n",
+        "For long, the big complaint about TensorFlow was *It's not flexible for debugging!* With the advent of TensorFlow 2.0, that changed drastically.\n",
+        "\n",
+        "Now TensorFlow allows you to run the oprations **eagerly**. That means, you can run TensorFlow operations by Python and return the outputs to Python again. That creates a lot of flexibility, especially for debugging. \n",
+        "\n",
+        "But there are some merits in NOT using the eagerly option. You can run operations on TensorFlow graphs that in some scenarios leads to significant speed up. According to TensorFlow:\n",
+        "\n",
+        "> Graphs are data structures that contain a set of [tf.Operation](https://www.tensorflow.org/api_docs/python/tf/Operation) objects, which represent units of computation; and [tf.Tensor](https://www.tensorflow.org/api_docs/python/tf/Tensor) objects, which represent the units of data that flow between operations. They are defined in a [tf.Graph](https://www.tensorflow.org/api_docs/python/tf/Graph) context. Since these graphs are data structures, they can be saved, run, and restored all without the original Python code.\n",
+        "\n",
+        "Let's have some example for transforming functions to graphs!\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "metadata": {
+        "id": "iJoLp_aUvBFR"
+      },
+      "source": [
+        "# Loading necessary libraries\n",
+        "import tensorflow as tf\n",
+        "import numpy as np\n",
+        "import timeit"
+      ],
+      "execution_count": 1,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "8a9c8qUj-MSj"
+      },
+      "source": [
+        "### Operation\n",
+        "\n",
+        "We can take a Python function on graph with [@tf.function](https://www.tensorflow.org/api_docs/python/tf/function) decorator."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "metadata": {
+        "id": "V-ugyJGRHc7S",
+        "outputId": "0bc0f944-4f6b-4fc2-95e2-8f517141cac6",
+        "colab": {
+          "base_uri": "https://localhost:8080/",
+          "height": 54
+        }
+      },
+      "source": [
+        "@tf.function\n",
+        "def multiply_fn(a, b):\n",
+        "  return tf.matmul(a, b)\n",
+        "\n",
+        "# Create some tensors\n",
+        "a = tf.constant([[0.5, 0.5]])\n",
+        "b = tf.constant([[10.0], [1.0]])\n",
+        "\n",
+        "# Check function\n",
+        "print('Multiple a of shape {} with b of shape {}'.format(a.shape, b.shape))\n",
+        "print(multiply_fn(a, b).numpy())\n"
+      ],
+      "execution_count": 2,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "text": [
+            "Multiple a of shape (1, 2) with b of shape (2, 1)\n",
+            "[[5.5]]\n"
+          ],
+          "name": "stdout"
+        }
+      ]
+    },
+    {
+      "cell_type": "code",
+      "metadata": {
+        "id": "zoh1CmJ59n-N",
+        "outputId": "addb1205-94a9-4f7b-d5b1-882ebed24a8a",
+        "colab": {
+          "base_uri": "https://localhost:8080/",
+          "height": 72
+        }
+      },
+      "source": [
+        "# Function without neing take to graph, i.e., with eager execution.\n",
+        "def add_fn(a, b):\n",
+        "  return tf.add(a, b)\n",
+        "\n",
+        "# Create some tensors\n",
+        "a = tf.constant([[0.5, 0.5]])\n",
+        "b = tf.constant([[10.0], [1.0]])\n",
+        "\n",
+        "# Check function\n",
+        "print('Add a of shape {} with b of shape {}'.format(a.shape, b.shape))\n",
+        "print(add_fn(a, b).numpy())"
+      ],
+      "execution_count": 3,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "text": [
+            "Add a of shape (1, 2) with b of shape (2, 1)\n",
+            "[[10.5 10.5]\n",
+            " [ 1.5  1.5]]\n"
+          ],
+          "name": "stdout"
+        }
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "j-WywaJU_MXR"
+      },
+      "source": [
+        "### Speedup\n",
+        "\n",
+        "Now let's define a custom model and run it:\n",
+        "\n",
+        "1. eagerly\n",
+        "2. on graph\n",
+        "\n",
+        "To check how to define models refer to: https://www.tensorflow.org/api_docs/python/tf/keras/Model"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "metadata": {
+        "id": "ynGAYcDC-y2W",
+        "outputId": "98d5fb24-d34b-4c53-85a9-b7456cfca4da",
+        "colab": {
+          "base_uri": "https://localhost:8080/",
+          "height": 90
+        }
+      },
+      "source": [
+        "class ModelShallow(tf.keras.Model):\n",
+        "\n",
+        "  def __init__(self):\n",
+        "    super(ModelShallow, self).__init__()\n",
+        "    self.dense1 = tf.keras.layers.Dense(10, activation=tf.nn.relu)\n",
+        "    self.dense2 = tf.keras.layers.Dense(20, activation=tf.nn.relu)\n",
+        "    self.dense3 = tf.keras.layers.Dense(30, activation=tf.nn.softmax)\n",
+        "    self.dropout = tf.keras.layers.Dropout(0.5)\n",
+        "\n",
+        "  def call(self, inputs, training=False):\n",
+        "    x = self.dense1(inputs)\n",
+        "    if training:\n",
+        "      x = self.dropout(x, training=training)\n",
+        "    x = self.dense2(x)\n",
+        "    out = self.dense3(x)\n",
+        "    return out\n",
+        "\n",
+        "class ModelDeep(tf.keras.Model):\n",
+        "\n",
+        "  def __init__(self):\n",
+        "    super(ModelDeep, self).__init__()\n",
+        "    self.dense1 = tf.keras.layers.Dense(1000, activation=tf.nn.relu)\n",
+        "    self.dense2 = tf.keras.layers.Dense(2000, activation=tf.nn.relu)\n",
+        "    self.dense3 = tf.keras.layers.Dense(3000, activation=tf.nn.softmax)\n",
+        "    self.dropout = tf.keras.layers.Dropout(0.5)\n",
+        "\n",
+        "  def call(self, inputs, training=False):\n",
+        "    x = self.dense1(inputs)\n",
+        "    if training:\n",
+        "      x = self.dropout(x, training=training)\n",
+        "    x = self.dense2(x)\n",
+        "    out = self.dense3(x)\n",
+        "    return out\n",
+        "\n",
+        "# Create the model with eager esxecution by default\n",
+        "model_shallow_with_eager = ModelShallow()\n",
+        "\n",
+        "# Take model to graph. \n",
+        "# NOTE: Instead of using decorators, we can ditectly operate tf.function on the model.\n",
+        "model_shallow_on_graph = tf.function(ModelShallow())\n",
+        "\n",
+        "# Model deep\n",
+        "model_deep_with_eager = ModelDeep()\n",
+        "model_deep_on_graph = tf.function(ModelDeep())\n",
+        "\n",
+        "# sample input\n",
+        "sample_input = tf.random.uniform([60, 28, 28])\n",
+        "\n",
+        "# Check time for shallow model\n",
+        "print(\"Shallow Model - Eager execution time:\", timeit.timeit(lambda: model_shallow_with_eager(sample_input), number=1000))\n",
+        "print(\"Shallow Model - Graph-based execution time:\", timeit.timeit(lambda: model_shallow_on_graph(sample_input), number=1000))\n",
+        "\n",
+        "# Check time for deep model\n",
+        "print(\"Deep Model - Eager execution time:\", timeit.timeit(lambda: model_deep_with_eager(sample_input), number=1000))\n",
+        "print(\"Deep Model - Graph-based execution time:\", timeit.timeit(lambda: model_deep_on_graph(sample_input), number=1000))"
+      ],
+      "execution_count": 4,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "text": [
+            "Shallow Model - Eager execution time: 2.758659444999921\n",
+            "Shallow Model - Graph-based execution time: 1.1618621510001503\n",
+            "Deep Model - Eager execution time: 477.634194022\n",
+            "Deep Model - Graph-based execution time: 460.01053104599987\n"
+          ],
+          "name": "stdout"
+        }
+      ]
+    },
+    {
+      "cell_type": "code",
+      "metadata": {
+        "id": "oBvht7tVAX4I"
+      },
+      "source": [
+        ""
+      ],
+      "execution_count": 4,
+      "outputs": []
+    }
+  ]
+}
\ No newline at end of file

codes/python/1-basics/graph.py

+# -*- coding: utf-8 -*-
+"""graph.ipynb
+
+Automatically generated by Colaboratory.
+
+Original file is located at
+    https://colab.research.google.com/drive/1ibfKtpxC_hIhZlPbefCoqpAS7jTdyiFw
+
+## Intorduction to TensorFlow graphs.
+
+For long, the big complaint about TensorFlow was *It's not flexible for debugging!* With the advent of TensorFlow 2.0, that changed drastically.
+
+Now TensorFlow allows you to run the oprations **eagerly**. That means, you can run TensorFlow operations by Python and return the outputs to Python again. That creates a lot of flexibility, especially for debugging. 
+
+But there are some merits in NOT using the eagerly option. You can run operations on TensorFlow graphs that in some scenarios leads to significant speed up. According to TensorFlow:
+
+> Graphs are data structures that contain a set of [tf.Operation](https://www.tensorflow.org/api_docs/python/tf/Operation) objects, which represent units of computation; and [tf.Tensor](https://www.tensorflow.org/api_docs/python/tf/Tensor) objects, which represent the units of data that flow between operations. They are defined in a [tf.Graph](https://www.tensorflow.org/api_docs/python/tf/Graph) context. Since these graphs are data structures, they can be saved, run, and restored all without the original Python code.
+
+Let's have some example for transforming functions to graphs!
+"""
+
+# Loading necessary libraries
+import tensorflow as tf
+import numpy as np
+import timeit
+
+"""### Operation
+
+We can take a Python function on graph with [@tf.function](https://www.tensorflow.org/api_docs/python/tf/function) decorator.
+"""
+
+@tf.function
+def multiply_fn(a, b):
+  return tf.matmul(a, b)
+
+# Create some tensors
+a = tf.constant([[0.5, 0.5]])
+b = tf.constant([[10.0], [1.0]])
+
+# Check function
+print('Multiple a of shape {} with b of shape {}'.format(a.shape, b.shape))
+print(multiply_fn(a, b).numpy())
+
+# Function without neing take to graph, i.e., with eager execution.
+def add_fn(a, b):
+  return tf.add(a, b)
+
+# Create some tensors
+a = tf.constant([[0.5, 0.5]])
+b = tf.constant([[10.0], [1.0]])
+
+# Check function
+print('Add a of shape {} with b of shape {}'.format(a.shape, b.shape))
+print(add_fn(a, b).numpy())
+
+"""### Speedup
+
+Now let's define a custom model and run it:
+
+1. eagerly
+2. on graph
+
+To check how to define models refer to: https://www.tensorflow.org/api_docs/python/tf/keras/Model
+"""
+
+class ModelShallow(tf.keras.Model):
+
+  def __init__(self):
+    super(ModelShallow, self).__init__()
+    self.dense1 = tf.keras.layers.Dense(10, activation=tf.nn.relu)
+    self.dense2 = tf.keras.layers.Dense(20, activation=tf.nn.relu)
+    self.dense3 = tf.keras.layers.Dense(30, activation=tf.nn.softmax)
+    self.dropout = tf.keras.layers.Dropout(0.5)
+
+  def call(self, inputs, training=False):
+    x = self.dense1(inputs)
+    if training:
+      x = self.dropout(x, training=training)
+    x = self.dense2(x)
+    out = self.dense3(x)
+    return out
+
+class ModelDeep(tf.keras.Model):
+
+  def __init__(self):
+    super(ModelDeep, self).__init__()
+    self.dense1 = tf.keras.layers.Dense(1000, activation=tf.nn.relu)
+    self.dense2 = tf.keras.layers.Dense(2000, activation=tf.nn.relu)
+    self.dense3 = tf.keras.layers.Dense(3000, activation=tf.nn.softmax)
+    self.dropout = tf.keras.layers.Dropout(0.5)
+
+  def call(self, inputs, training=False):
+    x = self.dense1(inputs)
+    if training:
+      x = self.dropout(x, training=training)
+    x = self.dense2(x)
+    out = self.dense3(x)
+    return out
+
+# Create the model with eager esxecution by default
+model_shallow_with_eager = ModelShallow()
+
+# Take model to graph. 
+# NOTE: Instead of using decorators, we can ditectly operate tf.function on the model.
+model_shallow_on_graph = tf.function(ModelShallow())
+
+# Model deep
+model_deep_with_eager = ModelDeep()
+model_deep_on_graph = tf.function(ModelDeep())
+
+# sample input
+sample_input = tf.random.uniform([60, 28, 28])
+
+# Check time for shallow model
+print("Shallow Model - Eager execution time:", timeit.timeit(lambda: model_shallow_with_eager(sample_input), number=1000))
+print("Shallow Model - Graph-based execution time:", timeit.timeit(lambda: model_shallow_on_graph(sample_input), number=1000))
+
+# Check time for deep model
+print("Deep Model - Eager execution time:", timeit.timeit(lambda: model_deep_with_eager(sample_input), number=1000))
+print("Deep Model - Graph-based execution time:", timeit.timeit(lambda: model_deep_on_graph(sample_input), number=1000))
+