README update

.gitattributes

-codes/* linguist-vendored
+codes/ipython/* linguist-vendored
 
 # Basic .gitattributes for a python repo.
 

README.rst

@@ -191,9 +191,9 @@ Neural Networks
 
 .. The tutorial in this section is just a simple entrance to TensorFlow.
 
-.. _welcomesourcecode: codes/0-welcome
+.. _welcomesourcecode: codes/python/0-welcome
 .. _Documentationcnnwelcome: docs/tutorials/0-welcome
-.. _ipythonwelcome: codes/0-welcome/code/0-welcome.ipynb
+.. _ipythonwelcome: codes/ipython/0-welcome/code/0-welcome.ipynb
 
 
 
@@ -208,12 +208,12 @@ Neural Networks
 .. ~~~~~~~~~~
 .. These tutorials are related to basics of TensorFlow.
 
-.. _basicmathsourcecode: codes/1-basics/basic_math_operations
+.. _basicmathsourcecode: codes/python/1-basics/basic_math_operations
 .. _Documentationbasicmath: docs/tutorials/1-basics/basic_math_operations
-.. _ipythonbasicmath: codes/1-basics/basic_math_operations/code/basic_math_operation.ipynb
+.. _ipythonbasicmath: codes/python/1-basics/basic_math_operations/code/basic_math_operation.ipynb
 
-.. _ipythonvariabls: codes/1-basics/variables/code/variables.ipynb
-.. _variablssourcecode: codes/1-basics/variables/README.rst
+.. _ipythonvariabls: codes/ipython/1-basics/variables/code/variables.ipynb
+.. _variablssourcecode: codes/python/1-basics/variables/README.rst
 .. _Documentationvariabls: docs/tutorials/1-basics/variables
 
 
@@ -230,20 +230,20 @@ Neural Networks
 .. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 .. We are going to present concepts of basic machine learning models and methods and show how to implement them in Tensorflow.
 
-.. _Linear Regression: codes/2-basics_in_machine_learning/linear_regression
-.. _LinearRegressionipython: codes/2-basics_in_machine_learning/linear_regression/code/linear_regression.ipynb
+.. _Linear Regression: codes/python/2-basics_in_machine_learning/linear_regression
+.. _LinearRegressionipython: codes/ipython/2-basics_in_machine_learning/linear_regression/code/linear_regression.ipynb
 .. _Documentationlr: docs/tutorials/2-basics_in_machine_learning/linear_regression
 
-.. _Logistic Regression: codes/2-basics_in_machine_learning/logistic_regression
-.. _LogisticRegressionipython: codes/2-basics_in_machine_learning/logistic_regression/code/logistic_regression.ipynb
+.. _Logistic Regression: codes/python/2-basics_in_machine_learning/logistic_regression
+.. _LogisticRegressionipython: codes//ipython/2-basics_in_machine_learning/logistic_regression/code/logistic_regression.ipynb
 .. _LogisticRegDOC: docs/tutorials/2-basics_in_machine_learning/logistic_regression
 
-.. _Linear SVM: codes/2-basics_in_machine_learning/linear_svm
-.. _LinearSVMipython: codes/2-basics_in_machine_learning/linear_svm/code/linear_svm.ipynb
+.. _Linear SVM: codes/python/2-basics_in_machine_learning/linear_svm
+.. _LinearSVMipython: codes//ipython/2-basics_in_machine_learning/linear_svm/code/linear_svm.ipynb
 
 
-.. _MultiClass Kernel SVM: codes/2-basics_in_machine_learning/multiclass_svm
-.. _MultiClassKernelSVMipython: codes/2-basics_in_machine_learning/multiclass_svm/code/multiclass_svm.ipynb
+.. _MultiClass Kernel SVM: codes/python/2-basics_in_machine_learning/multiclass_svm
+.. _MultiClassKernelSVMipython: codes/ipython/2-basics_in_machine_learning/multiclass_svm/code/multiclass_svm.ipynb
 
 
 .. +---+---------------------------------------------+----------------------------------------+
@@ -266,11 +266,11 @@ Neural Networks
 .. _Simple Convolutional Neural Networks: codes/3-neural_networks/convolutional-neural-network
 .. _Documentationcnn: docs/tutorials/3-neural_network/convolutiona_neural_network
 
-.. _Simple Multi Layer Perceptron: codes/3-neural_networks/multi-layer-perceptron
-.. _MultiLayerPerceptronipython: codes/3-neural_networks/multi-layer-perceptron/code/train_mlp.ipynb
+.. _Simple Multi Layer Perceptron: codes/python/3-neural_networks/multi-layer-perceptron
+.. _MultiLayerPerceptronipython: codes/ipython/3-neural_networks/multi-layer-perceptron/code/train_mlp.ipynb
 
-.. _RNN: codes/3-neural_networks/recurrent-neural-networks/code/rnn.py
-.. _RNNIpython: codes/3-neural_networks/recurrent-neural-networks/code/rnn.py
+.. _RNN: codes/python/3-neural_networks/recurrent-neural-networks/code/rnn.py
+.. _RNNIpython: codes/ipython/3-neural_networks/recurrent-neural-networks/code/rnn.ipynb
 
 
 .. +---+---------------------------------------------+----------------------------------------+

codes/python/0-welcome/README.rst

+
+===========================
+Welcome to TensorFlow World
+===========================
+
+This document is dedicated to explain how to run the python script for this tutorial.
+
+---------------------------
+Test TensorFlow Environment
+---------------------------
+
+``WARNING:`` If TensorFlow is installed in any environment(virtual environment, ...), it must be activated at first. So at first make sure the tensorFlow is available in the current environment using the following script:
+
+.. code:: shell
+
+    cd code/
+    python TensorFlow_Test.py
+    
+--------------------------------
+How to run the code in Terminal?
+--------------------------------
+
+    
+Please root to the ``code/`` directory and run the python script as the general form of below:
+
+.. code:: shell
+    
+    python [python_code_file.py] --log_dir='absolute/path/to/log_dir'
+    
+
+As an example the code can be executed as follows:
+
+.. code:: shell
+    
+    python 1-welcome.py --log_dir='~/log_dir'
+
+The ``--log_dir`` flag is to provide the address which the event files (for visualizing in Tensorboard) will be saved. The flag of ``--log_dir`` is not required because its default value is available in the source code as follows:
+
+.. code:: python
+    
+    tf.app.flags.DEFINE_string(
+    'log_dir', os.path.dirname(os.path.abspath(__file__)) + '/logs',
+    'Directory where event logs are written to.')
+
+----------------------------
+How to run the code in IDEs?
+----------------------------
+
+Since the code is ready-to-go, as long as the TensorFlow can be called in the IDE editor(Pycharm, Spyder,..), the code can be executed successfully.
+
+
+----------------------------
+How to run the Tensorboard?
+----------------------------
+.. _Google’s words: https://www.tensorflow.org/get_started/summaries_and_tensorboard
+TensorBoard is the graph visualization tools provided by TensorFlow. Using `Google’s words`_: “The computations you'll use TensorFlow for - like training a massive deep neural network - can be complex and confusing. To make it easier to understand,
+debug, and optimize TensorFlow programs, we've included a suite of visualization tools called
+TensorBoard.”
+
+The Tensorboard can be run as follows in the terminal:
+
+.. code:: shell
+    
+    tensorboard --logdir="absolute/path/to/log_dir"
+
+
+ 
+
+
+

codes/python/1-basics/basic_math_operations/README.rst

+
+===========================
+Welcome to TensorFlow World
+===========================
+
+This document is dedicated to explain how to run the python script for this tutorial.
+
+   
+--------------------------------
+How to run the code in Terminal?
+--------------------------------
+
+    
+Please root to the ``code/`` directory and run the python script as the general form of below:
+
+.. code:: shell
+    
+    python [python_code_file.py] --log_dir='absolute/path/to/log_dir'
+    
+
+As an example the code can be executed as follows:
+
+.. code:: shell
+    
+    python 1-welcome.py --log_dir='~/log_dir'
+
+The ``--log_dir`` flag is to provide the address which the event files (for visualizing in Tensorboard) will be saved. The flag of ``--log_dir`` is not required because its default value is available in the source code as follows:
+
+.. code:: python
+    
+    tf.app.flags.DEFINE_string(
+    'log_dir', os.path.dirname(os.path.abspath(__file__)) + '/logs',
+    'Directory where event logs are written to.')
+
+----------------------------
+How to run the code in IDEs?
+----------------------------
+
+Since the code is ready-to-go, as long as the TensorFlow can be called in the IDE editor(Pycharm, Spyder,..), the code can be executed successfully.
+
+
+----------------------------
+How to run the Tensorboard?
+----------------------------
+.. _Google’s words: https://www.tensorflow.org/get_started/summaries_and_tensorboard
+TensorBoard is the graph visualization tools provided by TensorFlow. Using `Google’s words`_: “The computations you'll use TensorFlow for - like training a massive deep neural network - can be complex and confusing. To make it easier to understand,
+debug, and optimize TensorFlow programs, we've included a suite of visualization tools called
+TensorBoard.”
+
+The Tensorboard can be run as follows in the terminal:
+
+.. code:: shell
+    
+    tensorboard --logdir="absolute/path/to/log_dir"
+
+
+ 
+
+
+

codes/python/1-basics/readme.rst

+==============================
+Basics
+==============================
+
+

codes/python/1-basics/variables/README.rst

+==========
+Variables
+==========
+
+This source code is dedicated to define and initialize variables.
+
+   
+--------------------------------
+How to run the code in Terminal?
+--------------------------------
+
+    
+Please root to the ``code/`` directory and run the python script as the general form of below:
+
+.. code:: shell
+    
+    python [python_code_file.py]
+    
+
+As an example the code can be executed as follows:
+
+.. code:: shell
+    
+    python variable.py
+
+----------------------------
+How to run the code in IDEs?
+----------------------------
+
+Since the code is ready-to-go, as long as the TensorFlow can be called in the IDE editor(Pycharm, Spyder,..), the code can be executed successfully.
+
+ 
+
+
+

codes/python/2-basics_in_machine_learning/linear_regression/README.rst

+==================
+Linear Regression
+==================
+
+This document is dedicated to explain how to run the python script for this tutorial. The documentation is available `here <Documentationlinearregression_>`_. Alternatively, you can check this ``Linear Regression using TensorFlow`` `blog post <blogpostlinearregression_>`_ for further details.
+
+.. _blogpostlinearregression: http://www.machinelearninguru.com/deep_learning/tensorflow/machine_learning_basics/linear_regresstion/linear_regression.html
+
+.. _Documentationlinearregression: https://github.com/astorfi/TensorFlow-World/wiki/Linear-Regeression
+
+-------------------
+Python Environment
+-------------------
+
+``WARNING:`` If TensorFlow is installed in any environment(virtual environment, ...), it must be activated at first. So at first make sure the tensorFlow is available in the current environment using the following script:
+
+--------------------------------
+How to run the code in Terminal?
+--------------------------------
+
+    
+Please root to the ``code/`` directory and run the python script as the general form of below:
+
+.. code:: shell
+    
+    python [python_code_file.py] 
+    
+
+As an example the code can be executed as follows:
+
+.. code:: shell
+    
+    python linear_regression.py --num_epochs=50
+
+The ``--num_epochs`` flag is to provide the number of epochs that will be used for training. The ``--num_epochs`` flag is not required because its default value is ``50`` and is provided in the source code as follows:
+
+.. code:: python
+    
+    tf.app.flags.DEFINE_integer(
+    'num_epochs', 50, 'The number of epochs for training the model. Default=50')
+
+----------------------------
+How to run the code in IDEs?
+----------------------------
+
+Since the code is ready-to-go, as long as the TensorFlow can be called in the IDE editor(Pycharm, Spyder,..), the code can be executed successfully.

codes/python/2-basics_in_machine_learning/linear_svm/README.rst

+===========
+Linear SVM
+===========
+
+This document is dedicated to explain how to run the python script for this tutorial. For this tutorial, we will create a linear SVM for separation of the data. The data that is used for this code is linearly separable.
+
+-------------------
+Python Environment
+-------------------
+
+``WARNING:`` If TensorFlow is installed in any environment(virtual environment, ...), it must be activated at first. So at first make sure the tensorFlow is available in the current environment using the following script:
+
+--------------------------------
+How to run the code in Terminal?
+--------------------------------
+
+    
+Please root to the ``code/`` directory and run the python script as the general form of below:
+
+.. code:: shell
+    
+    python [python_code_file.py] 
+    
+
+As an example the code can be executed as follows:
+
+.. code:: shell
+    
+    python linear_svm.py
+
+----------------------------
+How to run the code in IDEs?
+----------------------------
+
+Since the code is ready-to-go, as long as the TensorFlow can be called in the IDE editor(Pycharm, Spyder,..), the code can be executed successfully.

codes/python/2-basics_in_machine_learning/logistic_regression/README.rst

+==================
+Logistic Regression
+==================
+
+This document is dedicated to explaining how to run the python script for this tutorial. ``Logistic regression`` is a binary
+classification algorithm in which `yes` or `no` are the only possible responses. The linear output is transformed to a probability of course between zero and 1. The decision is made by thresholding the probability and saying it belongs to which class. We consider ``Softmax`` with ``cross entropy`` loss for minimizing the loss.
+
+-------------------
+Python Environment
+-------------------
+
+``WARNING:`` If TensorFlow is installed in any environment(virtual environment, ...), it must be activated at first. So at first make sure the tensorFlow is available in the current environment using the following script:
+
+--------------------------------
+How to run the code in Terminal?
+--------------------------------
+
+Please root to the ``code/`` directory and run the python script as the general form of below:
+
+.. code:: shell
+    
+    python [python_code_file.py] 
+    
+
+As an example the code can be executed as follows:
+
+.. code:: shell
+    
+    python logistic_regression.py --num_epochs=50 --batch_size=512 --max_num_checkpoint=10 --num_classes=2
+
+Different ``flags`` are provided for training. For the full list please refer to the source code. The above example is just an example as is!
+
+----------------------------
+How to run the code in IDEs?
+----------------------------
+
+Since the code is ready-to-go, as long as the TensorFlow can be called in the IDE editor(Pycharm, Spyder,..), the code can be executed successfully.

codes/python/2-basics_in_machine_learning/multiclass_svm/README.rst

+=======================
+Multi-Class Kernel SVM
+=======================
+
+This document is dedicated to explain how to run the python script for this tutorial. For this tutorial, we will create a Kernel SVM for separation of the data. The data that is used for this code is MNIST dataset. This document is inspired on `Implementing Multiclass SVMs <Multiclasssvm_>`_ open source code. However, in ours, we extend it to MNIST dataset and modify its method. 
+
+.. _Multiclasssvm: https://github.com/nfmcclure/tensorflow_cookbook/tree/master/04_Support_Vector_Machines/06_Implementing_Multiclass_SVMs
+
+
+-------------------
+Python Environment
+-------------------
+
+``WARNING:`` If TensorFlow is installed in any environment(virtual environment, ...), it must be activated at first. So at first make sure the tensorFlow is available in the current environment using the following script:
+
+--------------------------------
+How to run the code in Terminal?
+--------------------------------
+
+    
+Please root to the ``code/`` directory and run the python script as the general form of below:
+
+.. code:: shell
+    
+    python [python_code_file.py] 
+    
+
+As an example the code can be executed as follows:
+
+.. code:: shell
+    
+    python multiclass_SVM.py
+
+----------------------------
+How to run the code in IDEs?
+----------------------------
+
+Since the code is ready-to-go, as long as the TensorFlow can be called in the IDE editor(Pycharm, Spyder,..), the code can be executed successfully.

codes/python/3-neural_networks/convolutional-neural-network/code/auxiliary/progress_bar.py

+import sys
+
+
+def print_progress(progress, epoch_num, loss):
+    """
+    This function draw an active progress bar.
+    :param progress: Where we are:
+                       type: float
+                       value: [0,1]
+    :param epoch_num: number of epochs for training
+    :param loss: The loss for the specific batch in training phase.
+
+    :return: Progressing bar
+    """
+
+    # Define the length of bar
+    barLength = 30
+
+    # Ceck the input!
+    assert type(progress) is float, "id is not a float: %r" % id
+    assert 0 <= progress <= 1, "variable should be between zero and one!"
+
+    # Empty status while processing.
+    status = ""
+
+    # This part is to make a new line when the process is finished.
+    if progress >= 1:
+        progress = 1
+        status = "\r\n"
+
+    # Where we are in the progress!
+    indicator = int(round(barLength*progress))
+
+    # Print the appropriate progress phase!
+    list = [str(epoch_num), "#"*indicator , "-"*(barLength-indicator), progress*100, loss, status]
+    text = "\rEpoch {0[0]} {0[1]} {0[2]} %{0[3]:.2f} loss={0[4]:.3f} {0[5]}".format(list)
+    sys.stdout.write(text)
+    sys.stdout.flush()

codes/python/3-neural_networks/convolutional-neural-network/code/auxiliary/progress_bar.py~

+import sys
+
+
+def print_progress(progress, epoch_num, loss):
+    """
+    This function draw an active progress bar.
+    :param progress: Where we are:
+                       type: float
+                       value: [0,1]
+    :param epoch_num: number of epochs for training
+    :param loss: The loss for the specific batch in training phase.
+
+    :return: Progressing bar
+    """
+
+    # Define the length of bar
+    barLength = 30
+
+    # Ceck the input!
+    assert type(progress) is float, "id is not a float: %r" % id
+    assert 0 <= progress <= 1, "variable should be between zero and one!"
+
+    # Empty status while processing.
+    status = ""
+
+    # This part is to make a new line when the process is finished.
+    if progress >= 1:
+        progress = 1
+        status = "\r\n"
+
+    # Where we are in the progress!
+    indicator = int(round(barLength*progress))
+
+    # Print the appropriate progress phase!
+    list = [str(epoch_num), "#"*indicator , "-"*(barLength-indicator), progress*100, loss, status]
+    text = "\rEpoch {0[0]} {0[1]} {0[2]} %{0[3]:.2f} loss={0[4]:.3f} {0[5]}".format(list)
+    sys.stdout.write(text)
+    sys.stdout.flush()

codes/python/3-neural_networks/convolutional-neural-network/code/evaluation.sh

+
+# Run training.
+python test_classifier.py \
+  --batch_size=512 \
+  --allow_soft_placement
+

codes/python/3-neural_networks/convolutional-neural-network/code/input_function/input.py

+import numpy as np
+import collections
+
+
+class DATA_OBJECT(object):
+    def __init__(self,
+                 images,
+                 labels,
+                 num_classes=0,
+                 one_hot=False,
+                 dtype=np.float32,
+                 reshape=False):
+        """Data object construction.
+         images: The images of size [num_samples, rows, columns, depth].
+         labels: The labels of size [num_samples,]
+         num_classes: The number of classes in case one_hot labeling is desired.
+         one_hot=False: Turn the labels into one_hot format.
+         dtype=np.float32: The data type.
+         reshape=False: Reshape in case the feature vector extraction is desired.
+
+        """
+        # Define the date type.
+        if dtype not in (np.uint8, np.float32):
+            raise TypeError('Invalid image dtype %r, expected uint8 or float32' %
+                            dtype)
+        assert images.shape[0] == labels.shape[0], (
+            'images.shape: %s labels.shape: %s' % (images.shape, labels.shape))
+        self._num_samples = images.shape[0]
+
+        # [num_examples, rows, columns, depth] -> [num_examples, rows*columns]
+        if reshape:
+            assert images.shape[3] == 1
+            images = images.reshape(images.shape[0],
+                                    images.shape[1] * images.shape[2])
+
+        # Conver to float if necessary
+        if dtype == np.float32:
+            # Convert from [0, 255] -> [0.0, 1.0].
+            images = images.astype(dtype)
+            images = np.multiply(images, 1.0 / 255.0)
+        self._images = images
+        self._labels = labels
+
+        # If the one_hot flag is true, then the one_hot labeling supersedes the normal labeling.
+        if one_hot:
+            # If the one_hot labeling is desired, number of classes must be defined as one of the arguments of DATA_OBJECT class!
+            assert num_classes != 0, (
+                'You must specify the num_classes in the DATA_OBJECT for one_hot label construction!')
+
+            # Define the indexes.
+            index = np.arange(self._num_samples) * num_classes
+            one_hot_labels = np.zeros((self._num_samples, num_classes))
+            one_hot_labels.flat[index + labels.ravel()] = 1
+            self._labels = one_hot_labels
+
+    @property
+    def images(self):
+        return self._images
+
+    @property
+    def labels(self):
+        return self._labels
+
+    @property
+    def num_samples(self):
+        return self._num_samples
+
+
+def provide_data(mnist):
+    """
+    This function provide data object with desired shape.
+    The attribute of data object:
+        - train
+        - validation
+        - test
+    The sub attributs of the data object attributes:
+        -images
+        -labels
+
+    :param mnist: The downloaded MNIST dataset
+    :return: data: The data object.
+                   ex: data.train.images return the images of the dataset object in the training set!
+
+
+    """
+    ################################################
+    ########## Get the images and labels############
+    ################################################
+
+    # Note: This setup is specific to mnist data but can be generalized for any data.
+    # The ?_images(? can be train, validation or test) must have the format of [num_samples, rows, columns, depth] after extraction from data.
+    # The ?_labels(? can be train, validation or test) must have the format of [num_samples,] after extraction from data.
+    train_images = mnist.train.images
+    train_labels = mnist.train.labels
+    validation_images = mnist.validation.images
+    validation_labels = mnist.validation.labels
+    test_images = mnist.test.images
+    test_labels = mnist.test.labels
+
+    # Create separate objects for train, validation & test.
+    train = DATA_OBJECT(train_images, train_labels, num_classes=10, one_hot=True, dtype=np.float32, reshape=False)
+    validation = DATA_OBJECT(validation_images, validation_labels, num_classes=10, one_hot=True, dtype=np.float32,
+                             reshape=False)
+    test = DATA_OBJECT(test_images, test_labels, num_classes=10, one_hot=True, dtype=np.float32, reshape=False)
+
+    # Create the whole data object
+    DataSetObject = collections.namedtuple('DataSetObject', ['train', 'validation', 'test'])
+    data = DataSetObject(train=train, validation=validation, test=test)
+
+    return data

codes/python/3-neural_networks/convolutional-neural-network/code/net_structure/net.py

+#####################################
+# With some tiny modification, this code is the one used by Tensorflow slim at:
+# https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/slim
+# Please refer to the link for further explanations.
+
+### The difference is this architecture is written in fully-convolutional fashion.
+### The advantage is that, this model can be used for larger image sizes with some average pooling in the last layer.
+
+import tensorflow as tf
+
+slim = tf.contrib.slim
+
+
+def net_architecture(images, num_classes=10, is_training=False,
+                     dropout_keep_prob=0.5,
+                     spatial_squeeze=True,
+                     scope='Net'):
+    """Creates a variant of the Net model.
+
+    Args:
+      images: The batch of `Tensors`: size [batch_size, height, width, channels].
+      num_classes: Total number of classes.
+      is_training: Training/Validation.
+      dropout_keep_prob: The percentage of activation values: Only active in training mode!
+      scope: Variable_scope.
+
+    Returns:
+      logits: the pre-softmax activations of size [batch_size, `num_classes`]
+      end_points: The dictionary for the layers outputs.
+    """
+
+    # Create empty dictionary
+    end_points = {}
+
+    with tf.variable_scope(scope, 'Net', [images, num_classes]) as sc:
+        end_points_collection = sc.name + '_end_points'
+
+        # Collect outputs for conv2d and max_pool2d.
+        with tf.contrib.framework.arg_scope([tf.contrib.layers.conv2d, tf.contrib.layers.max_pool2d],
+                                            outputs_collections=end_points_collection):
+            # Layer-1
+            net = tf.contrib.layers.conv2d(images, 32, [5, 5], scope='conv1')
+            net = tf.contrib.layers.max_pool2d(net, [2, 2], 2, scope='pool1')
+
+            # Layer-2
+            net = tf.contrib.layers.conv2d(net, 64, [5, 5], scope='conv2')
+            net = tf.contrib.layers.max_pool2d(net, [2, 2], 2, scope='pool2')
+
+            # Layer-3
+            net = tf.contrib.layers.conv2d(net, 1024, [7, 7], padding='VALID', scope='fc3')
+            net = tf.contrib.layers.dropout(net, dropout_keep_prob, is_training=is_training,
+                                            scope='dropout3')
+
+            # Last layer which is the logits for classes
+            logits = tf.contrib.layers.conv2d(net, num_classes, [1, 1], activation_fn=None, scope='fc4')
+
+            # Return the collections as a dictionary
+            end_points = slim.utils.convert_collection_to_dict(end_points_collection)
+
+            # Squeeze spatially to eliminate extra dimensions.(embedding layer)
+            if spatial_squeeze:
+                logits = tf.squeeze(logits, [1, 2], name='fc4/squeezed')
+                end_points[sc.name + '/fc4'] = logits
+            return logits, end_points
+
+
+def net_arg_scope(weight_decay=0.0005, is_training=False):
+    """Defines the default network argument scope.
+
+    Args:
+      weight_decay: The weight decay to use for regularizing the model.
+
+    Returns:
+      An `arg_scope` to use for the model.
+    """
+    if is_training:
+        with tf.contrib.framework.arg_scope(
+                [tf.contrib.layers.conv2d],
+                padding='SAME',
+                weights_regularizer=slim.l2_regularizer(weight_decay),
+                weights_initializer=tf.contrib.layers.variance_scaling_initializer(factor=1.0, mode='FAN_AVG',
+                                                                                   uniform=False, seed=None,
+                                                                                   dtype=tf.float32),
+                activation_fn=tf.nn.relu) as sc:
+            return sc
+
+    else:
+        with tf.contrib.framework.arg_scope(
+                [tf.contrib.layers.conv2d],
+                padding='SAME',
+                activation_fn=tf.nn.relu) as sc:
+            return sc
+

codes/python/3-neural_networks/convolutional-neural-network/code/script/evaluation.sh

+# Where the logs will be saved to.
+test_dir=/home/sina/GITHUB/Tensorflow-Turorials/NeuralNetworks/convolutional-neural-network/code/test_logs
+
+# Where the checkpoints is saved to.
+checkpoint_dir=/home/sina/GITHUB/Tensorflow-Turorials/NeuralNetworks/convolutional-neural-network/code/checkpoints
+
+
+# Run training.
+python test_classifier.py \
+  --test_dir=${test_dir} \
+  --checkpoint_dir=${checkpoint_dir} \
+  --batch_size=512 \
+  --num_epochs=2 \
+  --allow_soft_placement
+

codes/python/3-neural_networks/convolutional-neural-network/code/script/train.sh

+# Where the logs will be saved to.
+train_dir=/home/sina/GITHUB/Tensorflow-Turorials/NeuralNetworks/convolutional-neural-network/code/train_logs
+
+# Where the checkpoints is saved to.
+checkpoint_dir=/home/sina/GITHUB/Tensorflow-Turorials/NeuralNetworks/convolutional-neural-network/code/checkpoints
+
+
+# Run training.
+python train_classifier.py \
+  --train_dir=${train_dir} \
+  --checkpoint_dir=${checkpoint_dir} \
+  --batch_size=512 \
+  --num_epochs=20 \
+  --max_num_checkpoint=10 \
+  --is_training \
+  --allow_soft_placement
+

codes/python/3-neural_networks/convolutional-neural-network/code/test_classifier.py

+from __future__ import print_function
+from tensorflow.examples.tutorials.mnist import input_data
+import tensorflow as tf
+import numpy as np
+from net_structure import net
+from input_function import input
+from auxiliary import progress_bar
+import os
+import sys
+
+######################################
+######### Necessary Flags ############
+######################################
+tf.app.flags.DEFINE_string(
+    'evaluation_path', os.path.dirname(os.path.abspath(__file__)) + '/test_log',
+    'Directory where event logs are written to.')
+
+tf.app.flags.DEFINE_string(
+    'checkpoints_directory',
+    os.path.dirname(os.path.abspath(__file__)) + '/checkpoints',
+    'Directory where checkpoints are written to.')
+
+tf.app.flags.DEFINE_integer('num_classes', 10,
+                            'Number of model clones to deploy.')
+
+tf.app.flags.DEFINE_integer('batch_size', np.power(2, 9),
+                            'Number of model clones to deploy.')
+
+#########################################
+########## status flags #################
+#########################################
+tf.app.flags.DEFINE_boolean('is_training', False,
+                            'Training/Testing.')
+
+tf.app.flags.DEFINE_boolean('allow_soft_placement', True,
+                            'Automatically put the variables on CPU if there is no GPU support.')
+
+tf.app.flags.DEFINE_boolean('log_device_placement', False,
+                            'Demonstrate which variables are on what device.')
+
+# Store all elemnts in FLAG structure!
+FLAGS = tf.app.flags.FLAGS
+
+################################################
+################# handling errors!##############
+################################################
+if not os.path.isabs(FLAGS.checkpoints_directory):
+    raise ValueError('You must assign absolute path for --checkpoints_directory')
+
+##########################################
+####### Load and Organize Data ###########
+##########################################
+'''
+In this part the input must be prepared.
+
+   1 - The MNIST data will be downloaded.
+   2 - The images and labels for both training and testing will be extracted.
+   3 - The prepared data format(?,784) is different by the appropriate image shape(?,28,28,1) which needs
+        to be fed to the CNN architecture. So it needs to be reshaped.
+
+'''
+
+# Download and get MNIST dataset(available in tensorflow.contrib.learn.python.learn.datasets.mnist)
+# It checks and download MNIST if it's not already downloaded then extract it.
+# The 'reshape' is True by default to extract feature vectors but we set it to false to we get the original images.
+mnist = input_data.read_data_sets("MNIST_data/", reshape=False, one_hot=False)
+
+# The 'input.provide_data' is provided to organize any custom dataset which has specific characteristics.
+data = input.provide_data(mnist)
+
+# Dimentionality of train
+dimensionality_train = data.train.images.shape
+
+# Dimensions
+num_train_samples = dimensionality_train[0]
+height = dimensionality_train[1]
+width = dimensionality_train[2]
+num_channels = dimensionality_train[3]
+
+#######################################
+########## Defining Graph ############
+#######################################
+
+graph = tf.Graph()
+with graph.as_default():
+    ###################################
+    ########### Parameters ############
+    ###################################
+
+    # global step
+    global_step = tf.Variable(0, name="global_step", trainable=False)
+
+    ###############################################
+    ########### Defining place holders ############
+    ###############################################
+    image_place = tf.placeholder(tf.float32, shape=([None, height, width, num_channels]), name='image')
+    label_place = tf.placeholder(tf.float32, shape=([None, FLAGS.num_classes]), name='gt')
+    dropout_parameter = tf.placeholder(tf.float32)
+
+    ##################################################
+    ########### Model + loss + accuracy ##############
+    ##################################################
+
+    # MODEL
+    joint_arg_scope = net.net_arg_scope(weight_decay=0.0005, is_training=FLAGS.is_training)
+    with tf.contrib.framework.arg_scope(joint_arg_scope):
+        logits_features, end_points = net.net_architecture(image_place, num_classes=FLAGS.num_classes,
+                                                  dropout_keep_prob=dropout_parameter,
+                                                  is_training=FLAGS.is_training)
+
+    # Define loss
+    with tf.name_scope('loss'):
+        loss_test = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=logits_features, labels=label_place))
+
+    # Accuracy
+    with tf.name_scope('accuracy_test'):
+        # Evaluate the model
+        correct_test_prediction = tf.equal(tf.argmax(logits_features, 1), tf.argmax(label_place, 1))
+
+        # Accuracy calculation
+        accuracy_test = tf.reduce_mean(tf.cast(correct_test_prediction, tf.float32))
+
+    ###############################################
+    ############ Define Sammaries #################
+    ###############################################
+
+    # Image summaries(draw three random images from data in both training and testing phases)
+    # The image summaries is only cerated for train summaries and it get three random images from the training set.
+    arr = np.random.randint(data.test.images.shape[0], size=(3,))
+    tf.summary.image('images', data.test.images[arr], max_outputs=3,
+                     collections=['per_epoch_train'])
+
+    # Histogram and scalar summaries sammaries
+    # sparsity: This summary is the fraction of zero activation for the output of each layer!
+    # activations: This summary is the histogram of activation for the output of each layer!
+    # WARNING: tf.summary.histogram can be very time consuming so it will be calculated per epoch!
+    for end_point in end_points:
+        x = end_points[end_point]
+        tf.summary.scalar('sparsity/' + end_point,
+                          tf.nn.zero_fraction(x), collections=['test'])
+
+    # Summaries for loss and accuracy
+    tf.summary.scalar("loss", loss_test, collections=['test'])
+    tf.summary.scalar("accuracy_test", accuracy_test, collections=['test'])
+    tf.summary.scalar("global_step", global_step, collections=['test'])
+
+    # Merge all summaries together.
+    summary_test_op = tf.summary.merge_all('test')
+
+    ########################################################
+    ############ # Defining the tensors list ###############
+    ########################################################
+
+    tensors_key = ['loss_test', 'accuracy_test', 'global_step', 'image_place', 'label_place',
+                   'summary_test_op']
+    tensors_values = [loss_test, accuracy_test, global_step, image_place, label_place, summary_test_op]
+    tensors = dict(zip(tensors_key, tensors_values))
+
+    ############################################
+    ############ Run the Session ###############
+    ############################################
+    session_conf = tf.ConfigProto(
+        allow_soft_placement=FLAGS.allow_soft_placement,
+        log_device_placement=FLAGS.log_device_placement)
+    sess = tf.Session(graph=graph, config=session_conf)
+
+    with sess.as_default():
+
+        # The saver op.
+        saver = tf.train.Saver()
+
+        # Initialize all variables
+        sess.run(tf.global_variables_initializer())
+
+        ###################################################################
+        ########## Defining the summary writers for test ###########
+        ###################################################################
+
+        test_summary_dir = os.path.join(FLAGS.evaluation_path, "summaries", "test")
+        test_summary_writer = tf.summary.FileWriter(test_summary_dir)
+        test_summary_writer.add_graph(sess.graph)
+
+        # The prefix for checkpoint files
+        checkpoint_prefix = 'model'
+
+        # Restoring the saved weights.
+        saver.restore(sess, os.path.join(FLAGS.checkpoints_directory, checkpoint_prefix))
+        print("Model restored...")
+
+        ###################################################################
+        ########## Run the training and loop over the batches #############
+        ###################################################################
+        num_test_samples = data.test.images.shape[0]
+        total_batch_test = int(num_test_samples / FLAGS.batch_size)
+
+        # go through the batches
+        test_accuracy = 0
+        for batch_num in range(total_batch_test):
+            #################################################
+            ########## Get the training batches #############
+            #################################################
+
+            start_idx = batch_num * FLAGS.batch_size
+            end_idx = (batch_num + 1) * FLAGS.batch_size
+
+            # Fit training using batch data
+            test_batch_data, test_batch_label = data.test.images[start_idx:end_idx], data.test.labels[
+                                                                                     start_idx:end_idx]
+
+            ########################################
+            ########## Run the session #############
+            ########################################
+
+            # Run session and Calculate batch loss and accuracy
+            # When the tensor tensors['global_step'] is evaluated, it will be incremented by one.
+
+            test_batch_accuracy, batch_loss, test_summaries, test_step = sess.run(
+                [tensors['accuracy_test'], tensors['loss_test'], tensors['summary_test_op'],
+                 tensors['global_step']],
+                feed_dict={tensors['image_place']: test_batch_data,
+                           tensors['label_place']: test_batch_label})
+            test_accuracy += test_batch_accuracy
+
+            ########################################
+            ########## Write summaries #############
+            ########################################
+
+            # Write the summaries
+            test_summary_writer.add_summary(test_summaries, global_step=test_step)
+
+            # # Write the specific summaries for training phase.
+            # train_summary_writer.add_summary(train_image_summary, global_step=training_step)
+
+            #################################################
+            ########## Plot the progressive bar #############
+            #################################################
+
+            progress = float(batch_num + 1) / total_batch_test
+            progress_bar.print_progress(progress, epoch_num=1, loss=batch_loss)
+
+
+        ######################################################################
+        ########## Calculate the accuracy for the whole test set #############
+        ######################################################################
+        test_accuracy_total = test_accuracy / float(total_batch_test)
+        print("Testing Accuracy= " + \
+              "{:.5f}".format(test_accuracy_total))

codes/python/3-neural_networks/convolutional-neural-network/code/train.sh

+
+# Run training.
+python train_classifier.py \
+  --batch_size=512 \
+  --max_num_checkpoint=10 \
+  --num_classes=10 \
+  --num_epochs=1 \
+  --initial_learning_rate=0.001 \
+  --num_epochs_per_decay=1 \
+  --is_training=True \
+  --allow_soft_placement=True \
+  --fine_tuning=False \
+  --online_test=True \
+  --log_device_placement=False
+

codes/python/3-neural_networks/convolutional-neural-network/code/train_classifier.py

+from __future__ import print_function
+from tensorflow.examples.tutorials.mnist import input_data
+import tensorflow as tf
+import numpy as np
+from net_structure import net
+from input_function import input
+import os
+import train_evaluation
+
+######################################
+######### Necessary Flags ############
+######################################
+
+tf.app.flags.DEFINE_string(
+    'train_dir', os.path.dirname(os.path.abspath(__file__)) + '/train_logs',
+    'Directory where event logs are written to.')
+
+tf.app.flags.DEFINE_string(
+    'checkpoint_dir',
+    os.path.dirname(os.path.abspath(__file__)) + '/checkpoints',
+    'Directory where checkpoints are written to.')
+
+tf.app.flags.DEFINE_integer('max_num_checkpoint', 10,
+                            'Maximum number of checkpoints that TensorFlow will keep.')
+
+tf.app.flags.DEFINE_integer('num_classes', 10,
+                            'Number of model clones to deploy.')
+
+tf.app.flags.DEFINE_integer('batch_size', np.power(2, 9),
+                            'Number of model clones to deploy.')
+
+tf.app.flags.DEFINE_integer('num_epochs', 1,
+                            'Number of epochs for training.')
+
+##########################################
+######## Learning rate flags #############
+##########################################
+tf.app.flags.DEFINE_float('initial_learning_rate', 0.001, 'Initial learning rate.')
+
+tf.app.flags.DEFINE_float(
+    'learning_rate_decay_factor', 0.95, 'Learning rate decay factor.')
+
+tf.app.flags.DEFINE_float(
+    'num_epochs_per_decay', 1, 'Number of epoch pass to decay learning rate.')
+
+#########################################
+########## status flags #################
+#########################################
+tf.app.flags.DEFINE_boolean('is_training', False,
+                            'Training/Testing.')
+
+tf.app.flags.DEFINE_boolean('fine_tuning', False,
+                            'Fine tuning is desired or not?.')
+
+tf.app.flags.DEFINE_boolean('online_test', True,
+                            'Fine tuning is desired or not?.')
+
+tf.app.flags.DEFINE_boolean('allow_soft_placement', True,
+                            'Automatically put the variables on CPU if there is no GPU support.')
+
+tf.app.flags.DEFINE_boolean('log_device_placement', False,
+                            'Demonstrate which variables are on what device.')
+
+# Store all elemnts in FLAG structure!
+FLAGS = tf.app.flags.FLAGS
+
+
+################################################
+################# handling errors!##############
+################################################
+if not os.path.isabs(FLAGS.train_dir):
+    raise ValueError('You must assign absolute path for --train_dir')
+
+if not os.path.isabs(FLAGS.checkpoint_dir):
+    raise ValueError('You must assign absolute path for --checkpoint_dir')
+
+##########################################
+####### Load and Organize Data ###########
+##########################################
+'''
+In this part the input must be prepared.
+
+   1 - The MNIST data will be downloaded.
+   2 - The images and labels for both training and testing will be extracted.
+   3 - The prepared data format(?,784) is different by the appropriate image shape(?,28,28,1) which needs
+        to be fed to the CNN architecture. So it needs to be reshaped.
+
+'''
+
+# Download and get MNIST dataset(available in tensorflow.contrib.learn.python.learn.datasets.mnist)
+# It checks and download MNIST if it's not already downloaded then extract it.
+# The 'reshape' is True by default to extract feature vectors but we set it to false to we get the original images.
+mnist = input_data.read_data_sets("MNIST_data/", reshape=False, one_hot=False)
+
+# The 'input.provide_data' is provided to organize any custom dataset which has specific characteristics.
+data = input.provide_data(mnist)
+
+# Dimentionality of train
+dimensionality_train = data.train.images.shape
+
+# Dimensions
+num_train_samples = dimensionality_train[0]
+height = dimensionality_train[1]
+width = dimensionality_train[2]
+num_channels = dimensionality_train[3]
+
+#######################################
+########## Defining Graph ############
+#######################################
+
+graph = tf.Graph()
+with graph.as_default():
+    ###################################
+    ########### Parameters ############
+    ###################################
+
+    # global step
+    global_step = tf.Variable(0, name="global_step", trainable=False)
+
+    # learning rate policy
+    decay_steps = int(num_train_samples / FLAGS.batch_size *
+                      FLAGS.num_epochs_per_decay)
+    learning_rate = tf.train.exponential_decay(FLAGS.initial_learning_rate,
+                                               global_step,
+                                               decay_steps,
+                                               FLAGS.learning_rate_decay_factor,
+                                               staircase=True,
+                                               name='exponential_decay_learning_rate')
+
+    ###############################################
+    ########### Defining place holders ############
+    ###############################################
+    image_place = tf.placeholder(tf.float32, shape=([None, height, width, num_channels]), name='image')
+    label_place = tf.placeholder(tf.float32, shape=([None, FLAGS.num_classes]), name='gt')
+    dropout_param = tf.placeholder(tf.float32)
+
+    ##################################################
+    ########### Model + Loss + Accuracy ##############
+    ##################################################
+
+    # MODEL
+    arg_scope = net.net_arg_scope(weight_decay=0.0005, is_training=FLAGS.is_training)
+    with tf.contrib.framework.arg_scope(arg_scope):
+        logits, end_points = net.net_architecture(image_place, num_classes=FLAGS.num_classes,
+                                                  dropout_keep_prob=dropout_param,
+                                                  is_training=FLAGS.is_training)
+
+    # Define loss
+    with tf.name_scope('loss'):
+        loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=logits, labels=label_place))
+
+    # Accuracy
+    with tf.name_scope('accuracy'):
+        # Evaluate the model
+        correct_pred = tf.equal(tf.argmax(logits, 1), tf.argmax(label_place, 1))
+
+        # Accuracy calculation
+        accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))
+
+    #############################################
+    ########### training operation ##############
+    #############################################
+
+    # Define optimizer by its default values
+    optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)
+
+    # 'train_op' is a operation that is run for gradient update on parameters.
+    # Each execution of 'train_op' is a training step.
+    # By passing 'global_step' to the optimizer, each time that the 'train_op' is run, Tensorflow
+    # update the 'global_step' and increment it by one!
+
+    # gradient update.
+    with tf.name_scope('train'):
+        grads_and_vars = optimizer.compute_gradients(loss)
+        train_op = optimizer.apply_gradients(grads_and_vars, global_step=global_step)
+
+    ###############################################
+    ############ Define Sammaries #################
+    ###############################################
+
+    # Image summaries(draw three random images from data in both training and testing phases)
+    # The image summaries is only cerated for train summaries and it get three random images from the training set.
+    arr = np.random.randint(data.train.images.shape[0], size=(3,))
+    tf.summary.image('images', data.train.images[arr], max_outputs=3,
+                     collections=['per_epoch_train'])
+
+    # Histogram and scalar summaries sammaries
+    # sparsity: This summary is the fraction of zero activation for the output of each layer!
+    # activations: This summary is the histogram of activation for the output of each layer!
+    # WARNING: tf.summary.histogram can be very time consuming so it will be calculated per epoch!
+    for end_point in end_points:
+        x = end_points[end_point]
+        tf.summary.scalar('sparsity/' + end_point,
+                          tf.nn.zero_fraction(x), collections=['train', 'test'])
+        tf.summary.histogram('activations/' + end_point, x, collections=['per_epoch_train'])
+
+    # Summaries for loss and accuracy
+    tf.summary.scalar("loss", loss, collections=['train', 'test'])
+    tf.summary.scalar("accuracy", accuracy, collections=['train', 'test'])
+    tf.summary.scalar("global_step", global_step, collections=['train'])
+    tf.summary.scalar("learning_rate", learning_rate, collections=['train'])
+
+    # Merge all summaries together.
+    summary_train_op = tf.summary.merge_all('train')
+    summary_test_op = tf.summary.merge_all('test')
+    summary_epoch_train_op = tf.summary.merge_all('per_epoch_train')
+
+    ########################################################
+    ############ # Defining the tensors list ###############
+    ########################################################
+
+    tensors_key = ['cost', 'accuracy', 'train_op', 'global_step', 'image_place', 'label_place', 'dropout_param',
+                   'summary_train_op', 'summary_test_op', 'summary_epoch_train_op']
+    tensors = [loss, accuracy, train_op, global_step, image_place, label_place, dropout_param, summary_train_op,
+               summary_test_op, summary_epoch_train_op]
+    tensors_dictionary = dict(zip(tensors_key, tensors))
+
+    ############################################
+    ############ Run the Session ###############
+    ############################################
+    session_conf = tf.ConfigProto(
+        allow_soft_placement=FLAGS.allow_soft_placement,
+        log_device_placement=FLAGS.log_device_placement)
+    sess = tf.Session(graph=graph, config=session_conf)
+
+    with sess.as_default():
+        # Run the saver.
+        # 'max_to_keep' flag determines the maximum number of models that the tensorflow save and keep. default by TensorFlow = 5.
+        saver = tf.train.Saver(max_to_keep=FLAGS.max_num_checkpoint)
+
+        # Initialize all variables
+        sess.run(tf.global_variables_initializer())
+
+        ###################################################
+        ############ Training / Evaluation ###############
+        ###################################################
+        train_evaluation.train(sess=sess, saver=saver, tensors=tensors_dictionary, data=data,
+                               train_dir=FLAGS.train_dir,
+                               finetuning=FLAGS.fine_tuning, online_test=FLAGS.online_test,
+                               num_epochs=FLAGS.num_epochs, checkpoint_dir=FLAGS.checkpoint_dir,
+                               batch_size=FLAGS.batch_size)
+
+        # Test in the end of experiment.
+        train_evaluation.evaluation(sess=sess, saver=saver, tensors=tensors_dictionary, data=data,
+                                    checkpoint_dir=FLAGS.checkpoint_dir)

codes/python/3-neural_networks/convolutional-neural-network/code/train_evaluation.py

+from __future__ import print_function
+import tensorflow as tf
+import numpy as np
+from auxiliary import progress_bar
+import os
+import sys
+
+
+def train(**keywords):
+    """
+    This function run the session whether in training or evaluation mode.
+    NOTE: **keywords is defined in order to make the code easily changable.
+    WARNING: All the arguments for the **keywords must be defined when calling this function.
+    **keywords:
+    :param sess: The default session.
+    :param saver: The saver operator to save and load the model weights.
+    :param tensors: The tensors dictionary defined by the graph.
+    :param data: The data structure.
+    :param train_dir: The training dir which is a reference for saving the logs and model checkpoints.
+    :param finetuning: If fine tuning should be done or random initialization is needed.
+    :param num_epochs: Number of epochs for training.
+    :param online_test: If the testing is done while training.
+    :param checkpoint_dir: The directory of the checkpoints.
+    :param batch_size: The training batch size.
+
+    :return:
+             Run the session.
+    """
+
+    # The prefix for checkpoint files
+    checkpoint_prefix = 'model'
+
+    ###################################################################
+    ########## Defining the summary writers for train/test ###########
+    ###################################################################
+
+    train_summary_dir = os.path.join(keywords['train_dir'], "summaries", "train")
+    train_summary_writer = tf.summary.FileWriter(train_summary_dir)
+    train_summary_writer.add_graph(keywords['sess'].graph)
+
+    test_summary_dir = os.path.join(keywords['train_dir'], "summaries", "test")
+    test_summary_writer = tf.summary.FileWriter(test_summary_dir)
+    test_summary_writer.add_graph(keywords['sess'].graph)
+
+    # If fie-tuning flag in 'True' the model will be restored.
+    if keywords['finetuning']:
+        keywords['saver'].restore(keywords['sess'], os.path.join(keywords['checkpoint_dir'], checkpoint_prefix))
+        print("Model restored for fine-tuning...")
+
+    ###################################################################
+    ########## Run the training and loop over the batches #############
+    ###################################################################
+    for epoch in range(keywords['num_epochs']):
+        total_batch_training = int(keywords['data'].train.images.shape[0] / keywords['batch_size'])
+
+        # go through the batches
+        for batch_num in range(total_batch_training):
+            #################################################
+            ########## Get the training batches #############
+            #################################################
+
+            start_idx = batch_num * keywords['batch_size']
+            end_idx = (batch_num + 1) * keywords['batch_size']
+
+            # Fit training using batch data
+            train_batch_data, train_batch_label = keywords['data'].train.images[start_idx:end_idx], keywords[
+                                                                                                        'data'].train.labels[
+                                                                                                    start_idx:end_idx]
+
+            ########################################
+            ########## Run the session #############
+            ########################################
+
+            # Run optimization op (backprop) and Calculate batch loss and accuracy
+            # When the tensor tensors['global_step'] is evaluated, it will be incremented by one.
+            batch_loss, _, train_summaries, training_step = keywords['sess'].run(
+                [keywords['tensors']['cost'], keywords['tensors']['train_op'], keywords['tensors']['summary_train_op'],
+                 keywords['tensors']['global_step']],
+                feed_dict={keywords['tensors']['image_place']: train_batch_data,
+                           keywords['tensors']['label_place']: train_batch_label,
+                           keywords['tensors']['dropout_param']: 0.5})
+
+            ########################################
+            ########## Write summaries #############
+            ########################################
+
+            # Write the summaries
+            train_summary_writer.add_summary(train_summaries, global_step=training_step)
+
+            # # Write the specific summaries for training phase.
+            # train_summary_writer.add_summary(train_image_summary, global_step=training_step)
+
+            #################################################
+            ########## Plot the progressive bar #############
+            #################################################
+
+            progress = float(batch_num + 1) / total_batch_training
+            progress_bar.print_progress(progress, epoch_num=epoch + 1, loss=batch_loss)
+
+        # ################################################################
+        # ############ Summaries per epoch of training ###################
+        # ################################################################
+        summary_epoch_train_op = keywords['tensors']['summary_epoch_train_op']
+        train_epoch_summaries = keywords['sess'].run(summary_epoch_train_op,
+                                                     feed_dict={keywords['tensors']['image_place']: train_batch_data,
+                                                                keywords['tensors']['label_place']: train_batch_label,
+                                                                keywords['tensors']['dropout_param']: 1.0})
+
+        # Put the summaries to the train summary writer.
+        train_summary_writer.add_summary(train_epoch_summaries, global_step=training_step)
+
+        #####################################################
+        ########## Evaluation on the test data #############
+        #####################################################
+
+        if keywords['online_test']:
+            # WARNING: In this evaluation the whole test data is fed. In case the test data is huge this implementation
+            #          may lead to memory error. In presense of large testing samples, batch evaluation on testing is
+            #          recommended as in the training phase.
+            test_accuracy_epoch, test_summaries = keywords['sess'].run(
+                [keywords['tensors']['accuracy'], keywords['tensors']['summary_test_op']],
+                feed_dict={keywords['tensors']['image_place']: keywords['data'].test.images,
+                           keywords['tensors'][
+                               'label_place']: keywords['data'].test.labels,
+                           keywords['tensors'][
+                               'dropout_param']: 1.})
+            print("Epoch " + str(epoch + 1) + ", Testing Accuracy= " + \
+                  "{:.5f}".format(test_accuracy_epoch))
+
+            ###########################################################
+            ########## Write the summaries for test phase #############
+            ###########################################################
+
+            # Returning the value of global_step if necessary
+            current_step = tf.train.global_step(keywords['sess'], keywords['tensors']['global_step'])
+
+            # Add the couter of global step for proper scaling between train and test summuries.
+            test_summary_writer.add_summary(test_summaries, global_step=current_step)
+
+    ###########################################################
+    ############ Saving the model checkpoint ##################
+    ###########################################################
+
+    # # The model will be saved when the training is done.
+
+    # Create the path for saving the checkpoints.
+    if not os.path.exists(keywords['checkpoint_dir']):
+        os.makedirs(keywords['checkpoint_dir'])
+
+    # save the model
+    save_path = keywords['saver'].save(keywords['sess'], os.path.join(keywords['checkpoint_dir'], checkpoint_prefix))
+    print("Model saved in file: %s" % save_path)
+
+
+    ############################################################################
+    ########## Run the session for pur evaluation on the test data #############
+    ############################################################################
+
+
+def evaluation(**keywords):
+    # The prefix for checkpoint files
+    checkpoint_prefix = 'model'
+
+    # Get the input arguments
+    saver = keywords['saver']
+    sess = keywords['sess']
+    checkpoint_dir = keywords['checkpoint_dir']
+    data = keywords['data']
+    accuracy_tensor = keywords['tensors']['accuracy']
+    image_place = keywords['tensors']['image_place']
+    label_place = keywords['tensors']['label_place']
+    dropout_param = keywords['tensors']['dropout_param']
+
+
+    # Restoring the saved weights.
+    saver.restore(sess, os.path.join(checkpoint_dir, checkpoint_prefix))
+    print("Model restored...")
+
+    test_set = data.test.images
+    test_label = data.test.labels
+    # Evaluation of the model
+    test_accuracy = 100 * keywords['sess'].run(accuracy_tensor, feed_dict={
+        image_place: test_set,
+        label_place: test_label,
+        dropout_param: 1.})
+
+    print("Final Test Accuracy is %% %.2f" % test_accuracy)

codes/python/3-neural_networks/convolutional-neural-network/readme.rst

+==============================
+Convolutional Neural Network
+==============================
+
+This is the code repository for the blog post `Train a Convolutional Neural Network as a Classifier`__. Please refer to this `wiki page`__ for more details.
+
+.. _cnn_classifier_post: http://machinelearninguru.com/deep_learning/tensorflow/neural_networks/cnn_classifier/cnn_classifier.html
+__ cnn_classifier_post_
+
+.. _cnn_classifier_wiki: https://github.com/astorfi/TensorFlow-World/wiki/Convolutional-Neural-Networks
+__ cnn_classifier_wiki_
+
+
+
+
+
+
+--------
+Training
+--------
+
+**Train:**
+
+The training can be run using the **train.sh** `bash script` file using the following command:
+
+.. code-block:: bash
+
+   ./train.sh
+
+The bash script is as below:
+
+
+.. code-block:: bash
+
+   python train_classifier.py \
+     --batch_size=512 \
+     --max_num_checkpoint=10 \
+     --num_classes=10 \
+     --num_epochs=1 \
+     --initial_learning_rate=0.001 \
+     --num_epochs_per_decay=1 \
+     --is_training=True \
+     --allow_soft_placement=True \
+     --fine_tuning=False \
+     --online_test=True \
+     --log_device_placement=False
+
+**helper:**
+
+In order to realize that what are the parameters as input running the following command is recommended:
+
+.. code-block:: bash
+
+   python train_classifier.py --help
+
+
+In which `train_classifier.py` is the main file for running the training. The result of the above command will be as below:
+
+.. code-block:: bash
+
+  --train_dir TRAIN_DIR
+                        Directory where event logs are written to.
+  --checkpoint_dir CHECKPOINT_DIR
+                        Directory where checkpoints are written to.
+  --max_num_checkpoint MAX_NUM_CHECKPOINT
+                        Maximum number of checkpoints that TensorFlow will
+                        keep.
+  --num_classes NUM_CLASSES
+                        Number of model clones to deploy.
+  --batch_size BATCH_SIZE
+                        Number of model clones to deploy.
+  --num_epochs NUM_EPOCHS
+                        Number of epochs for training.
+  --initial_learning_rate INITIAL_LEARNING_RATE
+                        Initial learning rate.
+  --learning_rate_decay_factor LEARNING_RATE_DECAY_FACTOR
+                        Learning rate decay factor.
+  --num_epochs_per_decay NUM_EPOCHS_PER_DECAY
+                        Number of epoch pass to decay learning rate.
+  --is_training [IS_TRAINING]
+                        Training/Testing.
+  --fine_tuning [FINE_TUNING]
+                        Fine tuning is desired or not?.
+  --online_test [ONLINE_TEST]
+                        Fine tuning is desired or not?.
+  --allow_soft_placement [ALLOW_SOFT_PLACEMENT]
+                        Automatically put the variables on CPU if there is no
+                        GPU support.
+  --log_device_placement [LOG_DEVICE_PLACEMENT]
+                        Demonstrate which variables are on what device.
+
+
+-----------
+Evaluation
+-----------
+
+The evaluation will be run using the **evaluation.sh** `bash script` file using the following command:
+
+.. code-block:: bash
+
+   ./evaluation.sh
+
+

codes/python/3-neural_networks/convolutional-neural-network/readme.rst~

+==============================
+Convolutional Neural Network
+==============================
+
+This is the code repository for the blog post `Train a Convolutional Neural Network as a Classifier`__. Please refer to this `wiki page`__ for more details.
+
+.. _cnn_classifier_post: http://machinelearninguru.com/deep_learning/tensorflow/neural_networks/cnn_classifier/cnn_classifier.html
+__ cnn_classifier_post_
+
+.. _cnn_classifier_wiki: https://github.com/astorfi/TensorFlow-World/wiki/Convolutional-Neural-Networks
+__ cnn_classifier_wiki_
+
+
+
+
+
+
+--------
+Training
+--------
+
+**Train:**
+
+The traing can be run using the **train.sh** `bash script` file using the following command:
+
+.. code-block:: bash
+
+   ./train.sh
+
+The bash script is as below:
+
+
+.. code-block:: bash
+
+   python train_classifier.py \
+     --batch_size=512 \
+     --max_num_checkpoint=10 \
+     --num_classes=10 \
+     --num_epochs=1 \
+     --initial_learning_rate=0.001 \
+     --num_epochs_per_decay=1 \
+     --is_training=True \
+     --allow_soft_placement=True \
+     --fine_tuning=False \
+     --online_test=True \
+     --log_device_placement=False
+
+**helper:**
+
+In order to realize that what are the parameters as input running the following command is recommended:
+
+.. code-block:: bash
+
+   python train_classifier.py --help
+
+
+In which `train_classifier.py` is the main file for running the training. The result of the above command will be as below:
+
+.. code-block:: bash
+
+  --train_dir TRAIN_DIR
+                        Directory where event logs are written to.
+  --checkpoint_dir CHECKPOINT_DIR
+                        Directory where checkpoints are written to.
+  --max_num_checkpoint MAX_NUM_CHECKPOINT
+                        Maximum number of checkpoints that TensorFlow will
+                        keep.
+  --num_classes NUM_CLASSES
+                        Number of model clones to deploy.
+  --batch_size BATCH_SIZE
+                        Number of model clones to deploy.
+  --num_epochs NUM_EPOCHS
+                        Number of epochs for training.
+  --initial_learning_rate INITIAL_LEARNING_RATE
+                        Initial learning rate.
+  --learning_rate_decay_factor LEARNING_RATE_DECAY_FACTOR
+                        Learning rate decay factor.
+  --num_epochs_per_decay NUM_EPOCHS_PER_DECAY
+                        Number of epoch pass to decay learning rate.
+  --is_training [IS_TRAINING]
+                        Training/Testing.
+  --fine_tuning [FINE_TUNING]
+                        Fine tuning is desired or not?.
+  --online_test [ONLINE_TEST]
+                        Fine tuning is desired or not?.
+  --allow_soft_placement [ALLOW_SOFT_PLACEMENT]
+                        Automatically put the variables on CPU if there is no
+                        GPU support.
+  --log_device_placement [LOG_DEVICE_PLACEMENT]
+                        Demonstrate which variables are on what device.
+
+
+-----------
+Evaluation
+-----------
+
+The evaluation will be run using the **evaluation.sh** `bash script` file using the following command:
+
+.. code-block:: bash
+
+   ./evaluation.sh
+
+

codes/python/3-neural_networks/convolutional-neural-network/requirements.txt

+- TensorFLow 1.0
+- numpy

codes/python/3-neural_networks/multi-layer-perceptron/code/evaluation.sh

+
+# Run training.
+python test_classifier.py \
+  --batch_size=512 \
+  --allow_soft_placement
+

codes/python/3-neural_networks/multi-layer-perceptron/code/test_classifier.py

+from __future__ import print_function
+from tensorflow.examples.tutorials.mnist import input_data
+import tensorflow as tf
+import numpy as np
+import os
+import sys
+
+######################################
+######### Necessary Flags ############
+######################################
+tf.app.flags.DEFINE_string(
+    'test_dir', os.path.dirname(os.path.abspath(__file__)) + '/test_logs',
+    'Directory where event logs are written to.')
+
+tf.app.flags.DEFINE_string(
+    'checkpoint_dir',
+    os.path.dirname(os.path.abspath(__file__)) + '/checkpoints',
+    'Directory where checkpoints are written to.')
+
+tf.app.flags.DEFINE_integer('num_classes', 10,
+                            'Number of model clones to deploy.')
+
+tf.app.flags.DEFINE_integer('batch_size', np.power(2, 9),
+                            'Number of model clones to deploy.')
+
+#########################################
+########## status flags #################
+#########################################
+tf.app.flags.DEFINE_boolean('is_training', False,
+                            'Training/Testing.')
+
+tf.app.flags.DEFINE_boolean('allow_soft_placement', True,
+                            'Automatically put the variables on CPU if there is no GPU support.')
+
+tf.app.flags.DEFINE_boolean('log_device_placement', False,
+                            'Demonstrate which variables are on what device.')
+
+# Store all elemnts in FLAG structure!
+FLAGS = tf.app.flags.FLAGS
+
+################################################
+################# handling errors!##############
+################################################
+if not os.path.isabs(FLAGS.checkpoint_dir):
+    raise ValueError('You must assign absolute path for --checkpoint_dir')
+
+##########################################
+####### Load and Organize Data ###########
+##########################################
+'''
+In this part the input must be prepared.
+
+   1 - The MNIST data will be downloaded.
+   2 - The images and labels for both training and testing will be extracted.
+   3 - The prepared data format(?,784) is different by the appropriate image shape(?,28,28,1) which needs
+        to be fed to the CNN architecture. So it needs to be reshaped.
+
+'''
+
+# Download and get MNIST dataset(available in tensorflow.contrib.learn.python.learn.datasets.mnist)
+# It checks and download MNIST if it's not already downloaded then extract it.
+# The 'reshape' is True by default to extract feature vectors but we set it to false to we get the original images.
+mnist = input_data.read_data_sets("MNIST_data/", reshape=True, one_hot=True)
+
+
+# Dimentionality of train
+dimensionality = mnist.train.images.shape
+
+# Dimensions
+num_train_samples = dimensionality[0]
+num_features = dimensionality[1]
+
+#######################################
+########## Defining Graph ############
+#######################################
+
+graph = tf.Graph()
+with graph.as_default():
+    ###################################
+    ########### Parameters ############
+    ###################################
+
+    # global step
+    global_step = tf.Variable(0, name="global_step", trainable=False)
+
+    ###############################################
+    ########### Defining place holders ############
+    ###############################################
+    image_place = tf.placeholder(tf.float32, shape=([None, num_features]), name='image')
+    label_place = tf.placeholder(tf.float32, shape=([None, FLAGS.num_classes]), name='gt')
+    dropout_param = tf.placeholder(tf.float32)
+
+    ##################################################
+    ########### Model + loss + accuracy ##############
+    ##################################################
+
+    # MODEL(MPL with two hidden layer)
+
+    # LAYER-1
+    net = tf.contrib.layers.fully_connected(inputs=image_place, num_outputs=250, scope='fc-1')
+
+    # LAYER-2
+    net = tf.contrib.layers.fully_connected(inputs=net, num_outputs=250, scope='fc-2')
+
+    # SOFTMAX
+    logits_last = tf.contrib.layers.fully_connected(inputs=net, num_outputs=FLAGS.num_classes, scope='fc-3')
+
+    # Define loss
+    loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=logits_last, labels=label_place))
+
+    # Accuracy
+    # Evaluate the model
+    pred_classifier = tf.equal(tf.argmax(logits_last, 1), tf.argmax(label_place, 1))
+
+    # Accuracy calculation
+    accuracy = tf.reduce_mean(tf.cast(pred_classifier, tf.float32))
+
+    ###############################################
+    ############ Define Sammaries #################
+    ###############################################
+
+    # Image summaries(draw three random images from data in both training and testing phases)
+    # The image summaries is only cerated for train summaries and it get three random images from the training set.
+    arr = np.random.randint(mnist.test.images.shape[0], size=(3,))
+    tf.summary.image('images', mnist.test.images[arr], max_outputs=3,
+                     collections=['per_epoch_train'])
+
+
+    # Summaries for loss and accuracy
+    tf.summary.scalar("loss", loss, collections=['test'])
+    tf.summary.scalar("accuracy", accuracy, collections=['test'])
+    tf.summary.scalar("global_step", global_step, collections=['test'])
+
+    # Merge all summaries together.
+    summary_test_op = tf.summary.merge_all('test')
+
+    ########################################################
+    ############ # Defining the tensors list ###############
+    ########################################################
+
+    # tensors_key = ['loss', 'accuracy', 'global_step', 'image_place', 'label_place',
+    #                'summary_test_op']
+    # tensors_values = [loss, accuracy, global_step, image_place, label_place, summary_test_op]
+    # tensors = dict(zip(tensors_key, tensors_values))
+
+    ############################################
+    ############ Run the Session ###############
+    ############################################
+    session_conf = tf.ConfigProto(
+        allow_soft_placement=FLAGS.allow_soft_placement,
+        log_device_placement=FLAGS.log_device_placement)
+    sess = tf.Session(graph=graph, config=session_conf)
+
+    with sess.as_default():
+
+        # The saver op.
+        saver = tf.train.Saver()
+
+        # Initialize all variables
+        sess.run(tf.global_variables_initializer())
+
+        ###################################################################
+        ########## Defining the summary writers for test ###########
+        ###################################################################
+
+        test_summary_dir = os.path.join(FLAGS.test_dir, "summaries", "test")
+        test_summary_writer = tf.summary.FileWriter(test_summary_dir)
+        test_summary_writer.add_graph(sess.graph)
+
+        # The prefix for checkpoint files
+        checkpoint_prefix = 'model'
+
+        # Restoring the saved weights.
+        saver.restore(sess, os.path.join(FLAGS.checkpoint_dir, checkpoint_prefix))
+        print("Model restored...")
+
+        ###################################################################
+        ########## Run the training and loop over the batches #############
+        ###################################################################
+        total_batch_test = int(mnist.test.images.shape[0] / FLAGS.batch_size)
+
+        # go through the batches
+        test_accuracy = 0
+        for batch_num in range(total_batch_test):
+            #################################################
+            ########## Get the training batches #############
+            #################################################
+
+            start_idx = batch_num * FLAGS.batch_size
+            end_idx = (batch_num + 1) * FLAGS.batch_size
+
+            # Fit training using batch data
+            test_batch_data, test_batch_label = mnist.test.images[start_idx:end_idx], mnist.test.labels[
+                                                                                     start_idx:end_idx]
+
+            ########################################
+            ########## Run the session #############
+            ########################################
+
+            # Run session and Calculate batch loss and accuracy
+            # When the tensor tensors['global_step'] is evaluated, it will be incremented by one.
+
+            test_batch_accuracy, batch_loss, test_summaries, test_step = sess.run(
+                [accuracy, loss, summary_test_op,
+                 global_step],
+                feed_dict={image_place: test_batch_data,
+                           label_place: test_batch_label})
+            test_accuracy += test_batch_accuracy
+
+            ########################################
+            ########## Write summaries #############
+            ########################################
+
+            # Write the summaries
+            test_summary_writer.add_summary(test_summaries, global_step=test_step)
+
+            # # Write the specific summaries for training phase.
+            # train_summary_writer.add_summary(train_image_summary, global_step=training_step)
+
+            #################################################
+            ########## Plot the progressive bar #############
+            #################################################
+
+            print("Batch " + str(batch_num + 1) + ", Testing Loss= " + \
+                  "{:.5f}".format(test_batch_accuracy))
+
+
+        ######################################################################
+        ########## Calculate the accuracy for the whole test set #############
+        ######################################################################
+        test_accuracy_total = test_accuracy / float(total_batch_test)
+        print("Total Test Accuracy= " + \
+              "{:.5f}".format(test_accuracy_total))

codes/python/3-neural_networks/multi-layer-perceptron/code/train.sh

+
+# Run training.
+python train_mlp.py \
+  --batch_size=512 \
+  --max_num_checkpoint=10 \
+  --num_classes=10 \
+  --num_epochs=1 \
+  --initial_learning_rate=0.001 \
+  --num_epochs_per_decay=1 \
+  --is_training=True \
+  --allow_soft_placement=True \
+  --fine_tuning=False \
+  --online_test=True \
+  --log_device_placement=False
+

codes/python/3-neural_networks/multi-layer-perceptron/code/train_mlp.ipynb

+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "from __future__ import print_function\n",
+    "from tensorflow.examples.tutorials.mnist import input_data\n",
+    "import tensorflow as tf\n",
+    "import numpy as np\n",
+    "import os"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "######################################\n",
+    "######### Necessary Flags ############\n",
+    "######################################\n",
+    "\n",
+    "num_classes = 10\n",
+    "batch_size = 128\n",
+    "num_epochs = 10"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "##########################################\n",
+    "######## Learning rate flags #############\n",
+    "##########################################\n",
+    "initial_learning_rate = 0.001\n",
+    "learning_rate_decay_factor = 0.95\n",
+    "num_epochs_per_decay = 1"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "#########################################\n",
+    "########## status flags #################\n",
+    "#########################################\n",
+    "is_training = False\n",
+    "fine_tuning = False\n",
+    "online_test = True\n",
+    "allow_soft_placement = True\n",
+    "log_device_placement = False"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Extracting MNIST_data/train-images-idx3-ubyte.gz\n",
+      "Extracting MNIST_data/train-labels-idx1-ubyte.gz\n",
+      "Extracting MNIST_data/t10k-images-idx3-ubyte.gz\n",
+      "Extracting MNIST_data/t10k-labels-idx1-ubyte.gz\n"
+     ]
+    }
+   ],
+   "source": [
+    "##########################################\n",
+    "####### Load and Organize Data ###########\n",
+    "##########################################\n",
+    "'''\n",
+    "In this part the input must be prepared.\n",
+    "\n",
+    "   1 - The MNIST data will be downloaded.\n",
+    "   2 - The images and labels for both training and testing will be extracted.\n",
+    "   3 - The prepared data format(?,784) is different by the appropriate image shape(?,28,28,1) which needs\n",
+    "        to be fed to the CNN architecture. So it needs to be reshaped.\n",
+    "\n",
+    "'''\n",
+    "\n",
+    "# Download and get MNIST dataset(available in tensorflow.contrib.learn.python.learn.datasets.mnist)\n",
+    "# It checks and download MNIST if it's not already downloaded then extract it.\n",
+    "# The 'reshape' is True by default to extract feature vectors but we set it to false to we get the original images.\n",
+    "mnist = input_data.read_data_sets(\"MNIST_data/\", reshape=True, one_hot=True)\n",
+    "train_data = mnist.train.images\n",
+    "train_label = mnist.train.labels\n",
+    "test_data = mnist.test.images\n",
+    "test_label = mnist.test.labels\n",
+    "\n",
+    "# # The 'input.provide_data' is provided to organize any custom dataset which has specific characteristics.\n",
+    "# data = input.provide_data(mnist)\n",
+    "\n",
+    "# Dimentionality of train\n",
+    "dimensionality_train = train_data.shape\n",
+    "\n",
+    "# Dimensions\n",
+    "num_train_samples = dimensionality_train[0]\n",
+    "num_features = dimensionality_train[1]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Epoch #1, Train Loss=0.248\n",
+      "Test Accuracy= 0.8565\n",
+      "Epoch #2, Train Loss=0.242\n",
+      "Test Accuracy= 0.8610\n",
+      "Epoch #3, Train Loss=0.240\n",
+      "Test Accuracy= 0.8676\n",
+      "Epoch #4, Train Loss=0.236\n",
+      "Test Accuracy= 0.8737\n",
+      "Epoch #5, Train Loss=0.235\n",
+      "Test Accuracy= 0.8740\n",
+      "Epoch #6, Train Loss=0.234\n",
+      "Test Accuracy= 0.8753\n",
+      "Epoch #7, Train Loss=0.234\n",
+      "Test Accuracy= 0.8756\n",
+      "Epoch #8, Train Loss=0.244\n",
+      "Test Accuracy= 0.8766\n",
+      "Epoch #9, Train Loss=0.234\n",
+      "Test Accuracy= 0.8765\n",
+      "Epoch #10, Train Loss=0.236\n",
+      "Test Accuracy= 0.8786\n",
+      "Final Test Accuracy is 0.88\n"
+     ]
+    }
+   ],
+   "source": [
+    "#######################################\n",
+    "########## Defining Graph ############\n",
+    "#######################################\n",
+    "\n",
+    "graph = tf.Graph()\n",
+    "with graph.as_default():\n",
+    "    ###################################\n",
+    "    ########### Parameters ############\n",
+    "    ###################################\n",
+    "\n",
+    "    # global step\n",
+    "    global_step = tf.Variable(0, name=\"global_step\", trainable=False)\n",
+    "\n",
+    "    # learning rate policy\n",
+    "    decay_steps = int(num_train_samples / batch_size *\n",
+    "                      num_epochs_per_decay)\n",
+    "    learning_rate = tf.train.exponential_decay(initial_learning_rate,\n",
+    "                                               global_step,\n",
+    "                                               decay_steps,\n",
+    "                                               learning_rate_decay_factor,\n",
+    "                                               staircase=True,\n",
+    "                                               name='exponential_decay_learning_rate')\n",
+    "\n",
+    "    ###############################################\n",
+    "    ########### Defining place holders ############\n",
+    "    ###############################################\n",
+    "    image_place = tf.placeholder(tf.float32, shape=([None, num_features]), name='image')\n",
+    "    label_place = tf.placeholder(tf.float32, shape=([None, num_classes]), name='gt')\n",
+    "    dropout_param = tf.placeholder(tf.float32)\n",
+    "\n",
+    "    ##################################################\n",
+    "    ########### Model + Loss + Accuracy ##############\n",
+    "    ##################################################\n",
+    "\n",
+    "    # MODEL(MPL with two hidden layer)\n",
+    "\n",
+    "    # LAYER-1\n",
+    "    net = tf.contrib.layers.fully_connected(inputs=image_place, num_outputs=250, scope='fc-1')\n",
+    "\n",
+    "    # LAYER-2\n",
+    "    net = tf.contrib.layers.fully_connected(inputs=net, num_outputs=250, scope='fc-2')\n",
+    "\n",
+    "    # SOFTMAX\n",
+    "    logits_pre_softmax = tf.contrib.layers.fully_connected(inputs=net, num_outputs=num_classes, scope='fc-3')\n",
+    "\n",
+    "    # Define loss\n",
+    "    softmax_loss = tf.reduce_mean(\n",
+    "        tf.nn.softmax_cross_entropy_with_logits(logits=logits_pre_softmax, labels=label_place))\n",
+    "\n",
+    "    # Accuracy\n",
+    "    accuracy = tf.reduce_mean(\n",
+    "        tf.cast(tf.equal(tf.argmax(logits_pre_softmax, 1), tf.argmax(label_place, 1)), tf.float32))\n",
+    "\n",
+    "    #############################################\n",
+    "    ########### training operation ##############\n",
+    "    #############################################\n",
+    "\n",
+    "    # Define optimizer by its default values\n",
+    "    optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)\n",
+    "\n",
+    "    # 'train_op' is a operation that is run for gradient update on parameters.\n",
+    "    # Each execution of 'train_op' is a training step.\n",
+    "    # By passing 'global_step' to the optimizer, each time that the 'train_op' is run, Tensorflow\n",
+    "    # update the 'global_step' and increment it by one!\n",
+    "\n",
+    "    # gradient update.\n",
+    "    with tf.name_scope('train_scope'):\n",
+    "        grads = optimizer.compute_gradients(softmax_loss)\n",
+    "        train_op = optimizer.apply_gradients(grads, global_step=global_step)\n",
+    "\n",
+    "    ###############################################\n",
+    "    ############ Define Sammaries #################\n",
+    "    ###############################################\n",
+    "\n",
+    "    # Summaries for loss and accuracy\n",
+    "    tf.summary.scalar(\"loss\", softmax_loss, collections=['train', 'test'])\n",
+    "    tf.summary.scalar(\"accuracy\", accuracy, collections=['train', 'test'])\n",
+    "    tf.summary.scalar(\"global_step\", global_step, collections=['train'])\n",
+    "    tf.summary.scalar(\"learning_rate\", learning_rate, collections=['train'])\n",
+    "\n",
+    "    # Merge all summaries together.\n",
+    "    summary_train_op = tf.summary.merge_all('train')\n",
+    "    summary_test_op = tf.summary.merge_all('test')\n",
+    "\n",
+    "    ############################################\n",
+    "    ############ Run the Session ###############\n",
+    "    ############################################\n",
+    "    session_conf = tf.ConfigProto(\n",
+    "        allow_soft_placement=allow_soft_placement,\n",
+    "        log_device_placement=log_device_placement)\n",
+    "    sess = tf.Session(graph=graph, config=session_conf)\n",
+    "\n",
+    "    with sess.as_default():\n",
+    "\n",
+    "        # Initialize all variables\n",
+    "        sess.run(tf.global_variables_initializer())\n",
+    "\n",
+    "        ###################################################################\n",
+    "        ########## Run the training and loop over the batches #############\n",
+    "        ###################################################################\n",
+    "        for epoch in range(num_epochs):\n",
+    "            total_batch_training = int(train_data.shape[0] / batch_size)\n",
+    "\n",
+    "            # go through the batches\n",
+    "            for batch_num in range(total_batch_training):\n",
+    "                #################################################\n",
+    "                ########## Get the training batches #############\n",
+    "                #################################################\n",
+    "\n",
+    "                start_idx = batch_num * batch_size\n",
+    "                end_idx = (batch_num + 1) * batch_size\n",
+    "\n",
+    "                # Fit training using batch data\n",
+    "                train_batch_data, train_batch_label = train_data[start_idx:end_idx], train_label[\n",
+    "                                                                                     start_idx:end_idx]\n",
+    "\n",
+    "                ########################################\n",
+    "                ########## Run the session #############\n",
+    "                ########################################\n",
+    "\n",
+    "                # Run optimization op (backprop) and Calculate batch loss and accuracy\n",
+    "                # When the tensor tensors['global_step'] is evaluated, it will be incremented by one.\n",
+    "                batch_loss, _, training_step = sess.run(\n",
+    "                    [softmax_loss, train_op, global_step],\n",
+    "                    feed_dict={image_place: train_batch_data,\n",
+    "                               label_place: train_batch_label,\n",
+    "                               dropout_param: 0.5})\n",
+    "\n",
+    "\n",
+    "                #################################################\n",
+    "                ########## Plot the progressive bar #############\n",
+    "                #################################################\n",
+    "\n",
+    "            print(\"Epoch #\" + str(epoch + 1) + \", Train Loss=\" + \\\n",
+    "                  \"{:.3f}\".format(batch_loss))\n",
+    "\n",
+    "            #####################################################\n",
+    "            ########## Evaluation on the test data #############\n",
+    "            #####################################################\n",
+    "\n",
+    "            if online_test:\n",
+    "                # WARNING: In this evaluation the whole test data is fed. In case the test data is huge this implementation\n",
+    "                #          may lead to memory error. In presense of large testing samples, batch evaluation on testing is\n",
+    "                #          recommended as in the training phase.\n",
+    "                test_accuracy_epoch, test_summaries = sess.run(\n",
+    "                    [accuracy, summary_test_op],\n",
+    "                    feed_dict={image_place: test_data,\n",
+    "                               label_place: test_label,\n",
+    "                               dropout_param: 1.})\n",
+    "                print(\"Test Accuracy= \" + \\\n",
+    "                      \"{:.4f}\".format(test_accuracy_epoch))\n",
+    "\n",
+    "                ###########################################################\n",
+    "                ########## Write the summaries for test phase #############\n",
+    "                ###########################################################\n",
+    "\n",
+    "                # Returning the value of global_step if necessary\n",
+    "                current_step = tf.train.global_step(sess, global_step)\n",
+    "\n",
+    "\n",
+    "        # Evaluation of the model\n",
+    "        total_test_accuracy = sess.run(accuracy, feed_dict={\n",
+    "            image_place: test_data,\n",
+    "            label_place: test_label,\n",
+    "            dropout_param: 1.})\n",
+    "\n",
+    "        print(\"Final Test Accuracy is %.2f\" % total_test_accuracy)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 2",
+   "language": "python",
+   "name": "python2"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 2
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython2",
+   "version": "2.7.13"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

codes/python/3-neural_networks/multi-layer-perceptron/code/train_mlp.py

+from __future__ import print_function
+from tensorflow.examples.tutorials.mnist import input_data
+import tensorflow as tf
+import numpy as np
+import os
+
+######################################
+######### Necessary Flags ############
+######################################
+
+tf.app.flags.DEFINE_string(
+    'train_root', os.path.dirname(os.path.abspath(__file__)) + '/train_logs',
+    'Directory where event logs are written to.')
+
+tf.app.flags.DEFINE_string(
+    'checkpoint_root',
+    os.path.dirname(os.path.abspath(__file__)) + '/checkpoints',
+    'Directory where checkpoints are written to.')
+
+tf.app.flags.DEFINE_integer('max_num_checkpoint', 10,
+                            'Maximum number of checkpoints that TensorFlow will keep.')
+
+tf.app.flags.DEFINE_integer('num_classes', 10,
+                            'Number of model clones to deploy.')
+
+tf.app.flags.DEFINE_integer('batch_size', np.power(2, 7),
+                            'Number of model clones to deploy.')
+
+tf.app.flags.DEFINE_integer('num_epochs', 5,
+                            'Number of epochs for training.')
+
+##########################################
+######## Learning rate flags #############
+##########################################
+tf.app.flags.DEFINE_float('initial_learning_rate', 0.001, 'Initial learning rate.')
+
+tf.app.flags.DEFINE_float(
+    'learning_rate_decay_factor', 0.95, 'Learning rate decay factor.')
+
+tf.app.flags.DEFINE_float(
+    'num_epochs_per_decay', 1, 'Number of epoch pass to decay learning rate.')
+
+#########################################
+########## status flags #################
+#########################################
+tf.app.flags.DEFINE_boolean('is_training', False,
+                            'Training/Testing.')
+
+tf.app.flags.DEFINE_boolean('fine_tuning', False,
+                            'Fine tuning is desired or not?.')
+
+tf.app.flags.DEFINE_boolean('online_test', True,
+                            'Fine tuning is desired or not?.')
+
+tf.app.flags.DEFINE_boolean('allow_soft_placement', True,
+                            'Automatically put the variables on CPU if there is no GPU support.')
+
+tf.app.flags.DEFINE_boolean('log_device_placement', False,
+                            'Demonstrate which variables are on what device.')
+
+# Store all elemnts in FLAG structure!
+FLAGS = tf.app.flags.FLAGS
+
+################################################
+################# handling errors!##############
+################################################
+if not os.path.isabs(FLAGS.train_root):
+    raise ValueError('You must assign absolute path for --train_root')
+
+if not os.path.isabs(FLAGS.checkpoint_root):
+    raise ValueError('You must assign absolute path for --checkpoint_root')
+
+##########################################
+####### Load and Organize Data ###########
+##########################################
+'''
+In this part the input must be prepared.
+
+   1 - The MNIST data will be downloaded.
+   2 - The images and labels for both training and testing will be extracted.
+   3 - The prepared data format(?,784) is different by the appropriate image shape(?,28,28,1) which needs
+        to be fed to the CNN architecture. So it needs to be reshaped.
+
+'''
+
+# Download and get MNIST dataset(available in tensorflow.contrib.learn.python.learn.datasets.mnist)
+# It checks and download MNIST if it's not already downloaded then extract it.
+# The 'reshape' is True by default to extract feature vectors but we set it to false to we get the original images.
+mnist = input_data.read_data_sets("MNIST_data/", reshape=True, one_hot=True)
+train_data = mnist.train.images
+train_label = mnist.train.labels
+test_data = mnist.test.images
+test_label = mnist.test.labels
+
+# # The 'input.provide_data' is provided to organize any custom dataset which has specific characteristics.
+# data = input.provide_data(mnist)
+
+# Dimentionality of train
+dimensionality_train = train_data.shape
+
+# Dimensions
+num_train_samples = dimensionality_train[0]
+num_features = dimensionality_train[1]
+
+#######################################
+########## Defining Graph ############
+#######################################
+
+graph = tf.Graph()
+with graph.as_default():
+    ###################################
+    ########### Parameters ############
+    ###################################
+
+    # global step
+    global_step = tf.Variable(0, name="global_step", trainable=False)
+
+    # learning rate policy
+    decay_steps = int(num_train_samples / FLAGS.batch_size *
+                      FLAGS.num_epochs_per_decay)
+    learning_rate = tf.train.exponential_decay(FLAGS.initial_learning_rate,
+                                               global_step,
+                                               decay_steps,
+                                               FLAGS.learning_rate_decay_factor,
+                                               staircase=True,
+                                               name='exponential_decay_learning_rate')
+
+    ###############################################
+    ########### Defining place holders ############
+    ###############################################
+    image_place = tf.placeholder(tf.float32, shape=([None, num_features]), name='image')
+    label_place = tf.placeholder(tf.float32, shape=([None, FLAGS.num_classes]), name='gt')
+    dropout_param = tf.placeholder(tf.float32)
+
+    ##################################################
+    ########### Model + Loss + Accuracy ##############
+    ##################################################
+
+    # MODEL(MPL with two hidden layer)
+
+    # LAYER-1
+    net = tf.contrib.layers.fully_connected(inputs=image_place, num_outputs=250, scope='fc-1')
+
+    # LAYER-2
+    net = tf.contrib.layers.fully_connected(inputs=net, num_outputs=250, scope='fc-2')
+
+    # SOFTMAX
+    logits_pre_softmax = tf.contrib.layers.fully_connected(inputs=net, num_outputs=FLAGS.num_classes, scope='fc-3')
+
+    # Define loss
+    softmax_loss = tf.reduce_mean(
+        tf.nn.softmax_cross_entropy_with_logits(logits=logits_pre_softmax, labels=label_place))
+
+    # Accuracy
+    accuracy = tf.reduce_mean(
+        tf.cast(tf.equal(tf.argmax(logits_pre_softmax, 1), tf.argmax(label_place, 1)), tf.float32))
+
+    #############################################
+    ########### training operation ##############
+    #############################################
+
+    # Define optimizer by its default values
+    optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)
+
+    # 'train_op' is a operation that is run for gradient update on parameters.
+    # Each execution of 'train_op' is a training step.
+    # By passing 'global_step' to the optimizer, each time that the 'train_op' is run, Tensorflow
+    # update the 'global_step' and increment it by one!
+
+    # gradient update.
+    with tf.name_scope('train_scope'):
+        grads = optimizer.compute_gradients(softmax_loss)
+        train_op = optimizer.apply_gradients(grads, global_step=global_step)
+
+    ###############################################
+    ############ Define Sammaries #################
+    ###############################################
+
+    # Summaries for loss and accuracy
+    tf.summary.scalar("loss", softmax_loss, collections=['train', 'test'])
+    tf.summary.scalar("accuracy", accuracy, collections=['train', 'test'])
+    tf.summary.scalar("global_step", global_step, collections=['train'])
+    tf.summary.scalar("learning_rate", learning_rate, collections=['train'])
+
+    # Merge all summaries together.
+    summary_train_op = tf.summary.merge_all('train')
+    summary_test_op = tf.summary.merge_all('test')
+
+    ############################################
+    ############ Run the Session ###############
+    ############################################
+    session_conf = tf.ConfigProto(
+        allow_soft_placement=FLAGS.allow_soft_placement,
+        log_device_placement=FLAGS.log_device_placement)
+    sess = tf.Session(graph=graph, config=session_conf)
+
+    with sess.as_default():
+        # Run the saver.
+        # 'max_to_keep' flag determines the maximum number of models that the tensorflow save and keep. default by TensorFlow = 5.
+        saver = tf.train.Saver(max_to_keep=FLAGS.max_num_checkpoint)
+
+        # Initialize all variables
+        sess.run(tf.global_variables_initializer())
+
+        ###################################################
+        ############ Training / Evaluation ###############
+        ###################################################
+
+        # The prefix for checkpoint files
+        checkpoint_prefix = 'model'
+
+        ###################################################################
+        ########## Defining the summary writers for train/test ###########
+        ###################################################################
+
+        train_summary_dir = os.path.join(FLAGS.train_root, "summaries", "train")
+        train_summary_writer = tf.summary.FileWriter(train_summary_dir)
+        train_summary_writer.add_graph(sess.graph)
+
+        test_summary_dir = os.path.join(FLAGS.train_root, "summaries", "test")
+        test_summary_writer = tf.summary.FileWriter(test_summary_dir)
+        test_summary_writer.add_graph(sess.graph)
+
+        # If fine-tuning flag in 'True' the model will be restored.
+        if FLAGS.fine_tuning:
+            saver.restore(sess, os.path.join(FLAGS.checkpoint_root, checkpoint_prefix))
+            print("Model restored for fine-tuning...")
+
+        ###################################################################
+        ########## Run the training and loop over the batches #############
+        ###################################################################
+        for epoch in range(FLAGS.num_epochs):
+            total_batch_training = int(train_data.shape[0] / FLAGS.batch_size)
+
+            # go through the batches
+            for batch_num in range(total_batch_training):
+                #################################################
+                ########## Get the training batches #############
+                #################################################
+
+                start_idx = batch_num * FLAGS.batch_size
+                end_idx = (batch_num + 1) * FLAGS.batch_size
+
+                # Fit training using batch data
+                train_batch_data, train_batch_label = train_data[start_idx:end_idx], train_label[
+                                                                                     start_idx:end_idx]
+
+                ########################################
+                ########## Run the session #############
+                ########################################
+
+                # Run optimization op (backprop) and Calculate batch loss and accuracy
+                # When the tensor tensors['global_step'] is evaluated, it will be incremented by one.
+                batch_loss, _, train_summaries, training_step = sess.run(
+                    [softmax_loss, train_op,
+                     summary_train_op,
+                     global_step],
+                    feed_dict={image_place: train_batch_data,
+                               label_place: train_batch_label,
+                               dropout_param: 0.5})
+
+                ########################################
+                ########## Write summaries #############
+                ########################################
+
+                # Write the summaries
+                train_summary_writer.add_summary(train_summaries, global_step=training_step)
+
+                # # Write the specific summaries for training phase.
+                # train_summary_writer.add_summary(train_image_summary, global_step=training_step)
+
+                #################################################
+                ########## Plot the progressive bar #############
+                #################################################
+
+            print("Epoch #" + str(epoch + 1) + ", Train Loss=" + \
+                  "{:.3f}".format(batch_loss))
+
+            #####################################################
+            ########## Evaluation on the test data #############
+            #####################################################
+
+            if FLAGS.online_test:
+                # WARNING: In this evaluation the whole test data is fed. In case the test data is huge this implementation
+                #          may lead to memory error. In presense of large testing samples, batch evaluation on testing is
+                #          recommended as in the training phase.
+                test_accuracy_epoch, test_summaries = sess.run(
+                    [accuracy, summary_test_op],
+                    feed_dict={image_place: test_data,
+                               label_place: test_label,
+                               dropout_param: 1.})
+                print("Test Accuracy= " + \
+                      "{:.4f}".format(test_accuracy_epoch))
+
+                ###########################################################
+                ########## Write the summaries for test phase #############
+                ###########################################################
+
+                # Returning the value of global_step if necessary
+                current_step = tf.train.global_step(sess, global_step)
+
+                # Add the couter of global step for proper scaling between train and test summuries.
+                test_summary_writer.add_summary(test_summaries, global_step=current_step)
+
+        ###########################################################
+        ############ Saving the model checkpoint ##################
+        ###########################################################
+
+        # # The model will be saved when the training is done.
+
+        # Create the path for saving the checkpoints.
+        if not os.path.exists(FLAGS.checkpoint_root):
+            os.makedirs(FLAGS.checkpoint_root)
+
+        # save the model
+        save_path = saver.save(sess, os.path.join(FLAGS.checkpoint_root, checkpoint_prefix))
+        print("Model saved in file: %s" % save_path)
+
+        ############################################################################
+        ########## Run the session for pur evaluation on the test data #############
+        ############################################################################
+
+        # The prefix for checkpoint files
+        checkpoint_prefix = 'model'
+
+        # Restoring the saved weights.
+        saver.restore(sess, os.path.join(FLAGS.checkpoint_root, checkpoint_prefix))
+        print("Model restored...")
+
+        # Evaluation of the model
+        total_test_accuracy = sess.run(accuracy, feed_dict={
+            image_place: test_data,
+            label_place: test_label,
+            dropout_param: 1.})
+
+        print("Final Test Accuracy is %.2f" % total_test_accuracy)

codes/python/3-neural_networks/multi-layer-perceptron/readme.rst

+=========================
+Multi Layer Perceptron
+=========================
+
+This code is developed for training a ``Multi Layer Perceptron`` architecture in which the input will be feed-forwarded to the network that contains some hidden layers.
+
+.. figure:: https://github.com/astorfi/TensorFlow-World/blob/master/docs/_img/3-neural_network/multi-layer-perceptron/neural-network.png
+   :scale: 50 %
+   :align: center
+   
+
+--------
+Training
+--------
+
+**Train:**
+
+The training can be run using the **train.sh** `bash script` file using the following command:
+
+.. code-block:: bash
+
+   ./train.sh
+
+The bash script is as below:
+
+
+.. code-block:: bash
+
+   python train_mlp.py \
+     --batch_size=512 \
+     --max_num_checkpoint=10 \
+     --num_classes=10 \
+     --num_epochs=1 \
+     --initial_learning_rate=0.001 \
+     --num_epochs_per_decay=1 \
+     --is_training=True \
+     --allow_soft_placement=True \
+     --fine_tuning=False \
+     --online_test=True \
+     --log_device_placement=False
+
+**helper:**
+
+In order to realize that what are the parameters as input running the following command is recommended:
+
+.. code-block:: bash
+
+   python train_mlp.py --help
+
+
+In which `train_mlp.py` is the main file for running the training. The result of the above command will be as below:
+
+.. code-block:: bash
+
+  --train_dir TRAIN_DIR
+                        Directory where event logs are written to.
+  --checkpoint_dir CHECKPOINT_DIR
+                        Directory where checkpoints are written to.
+  --max_num_checkpoint MAX_NUM_CHECKPOINT
+                        Maximum number of checkpoints that TensorFlow will
+                        keep.
+  --num_classes NUM_CLASSES
+                        Number of model clones to deploy.
+  --batch_size BATCH_SIZE
+                        Number of model clones to deploy.
+  --num_epochs NUM_EPOCHS
+                        Number of epochs for training.
+  --initial_learning_rate INITIAL_LEARNING_RATE
+                        Initial learning rate.
+  --learning_rate_decay_factor LEARNING_RATE_DECAY_FACTOR
+                        Learning rate decay factor.
+  --num_epochs_per_decay NUM_EPOCHS_PER_DECAY
+                        Number of epoch pass to decay learning rate.
+  --is_training [IS_TRAINING]
+                        Training/Testing.
+  --fine_tuning [FINE_TUNING]
+                        Fine tuning is desired or not?.
+  --online_test [ONLINE_TEST]
+                        Fine tuning is desired or not?.
+  --allow_soft_placement [ALLOW_SOFT_PLACEMENT]
+                        Automatically put the variables on CPU if there is no
+                        GPU support.
+  --log_device_placement [LOG_DEVICE_PLACEMENT]
+                        Demonstrate which variables are on what device.
+
+
+-----------
+Evaluation
+-----------
+
+The evaluation will be run using the **evaluation.sh** `bash script` file using the following command:
+
+.. code-block:: bash
+
+   ./evaluation.sh
+
+

codes/python/3-neural_networks/multi-layer-perceptron/requirements.txt

+- TensorFLow 1.0
+- numpy

codes/python/3-neural_networks/readme.md

+The tutorial in TensorFlow related to Neural Network architectures.