{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <center/>同步训练和验证模型体验"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 概述"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在面对复杂网络时，往往需要进行几十甚至几百次的epoch训练。而在训练之前，往往很难掌握在训练到第几个epoch时，模型的精度能达到满足要求的程度。所以经常会采用一边训练的同时，在相隔固定epoch的位置对模型进行精度验证，并保存相应的模型，等训练完毕后，通过查看对应模型精度的变化就能迅速地挑选出相对最优的模型，本文将采用这种方法，以LeNet网络为样本，进行示例。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "整体流程如下：\n",
    "1. 数据集准备。\n",
    "2. 构建神经网络。\n",
    "3. 定义回调函数EvalCallBack。\n",
    "4. 定义训练网络并执行。\n",
    "5. 定义绘图函数并对不同epoch下的模型精度绘制出折线图。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 数据准备"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 数据集的下载"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "训练数据集下载地址：{\"http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz \", \"http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz \"}。\n",
    "\n",
    "测试数据集：{\"<http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz>\", \"<http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz>\"}\n",
    "<br/>数据集放在----*Jupyter工作目录+\\MNIST_Data\\*，如下图结构："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "MNIST\n",
    "├── test\n",
    "│   ├── t10k-images-idx3-ubyte\n",
    "│   └── t10k-labels-idx1-ubyte\n",
    "└── train\n",
    "    ├── train-images-idx3-ubyte\n",
    "    └── train-labels-idx1-ubyte \n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 数据集的增强操作"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下载下来后的数据集，需要通过`mindspore.dataset`处理成适用于MindSpore框架的数据，再使用一系列框架中提供的工具进行数据增强操作来适应LeNet网络的数据处理需求。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import mindspore.dataset as ds\n",
    "import mindspore.dataset.transforms.vision.c_transforms as CV\n",
    "import mindspore.dataset.transforms.c_transforms as C\n",
    "from mindspore.dataset.transforms.vision import Inter\n",
    "from mindspore.common import dtype as mstype\n",
    "\n",
    "def create_dataset(data_path, batch_size=32, repeat_size=1,\n",
    "                   num_parallel_workers=1):\n",
    "    # define dataset\n",
    "    mnist_ds = ds.MnistDataset(data_path)\n",
    "\n",
    "    # define map operations\n",
    "    resize_op = CV.Resize((32, 32), interpolation=Inter.LINEAR)  \n",
    "    rescale_nml_op = CV.Rescale(1 / 0.3081, -1 * 0.1307 / 0.3081) \n",
    "    rescale_op = CV.Rescale(1/255.0, 0.0) \n",
    "    hwc2chw_op = CV.HWC2CHW() \n",
    "    type_cast_op = C.TypeCast(mstype.int32) \n",
    "\n",
    "    # apply map operations on images\n",
    "    mnist_ds = mnist_ds.map(input_columns=\"label\", operations=type_cast_op, num_parallel_workers=num_parallel_workers)\n",
    "    mnist_ds = mnist_ds.map(input_columns=\"image\", operations=[resize_op,rescale_op,rescale_nml_op,hwc2chw_op],\n",
    "                            num_parallel_workers=num_parallel_workers)\n",
    "\n",
    "    # apply DatasetOps\n",
    "    buffer_size = 10000\n",
    "    mnist_ds = mnist_ds.shuffle(buffer_size=buffer_size)\n",
    "    mnist_ds = mnist_ds.batch(batch_size, drop_remainder=True)\n",
    "    mnist_ds = mnist_ds.repeat(repeat_size)\n",
    "    \n",
    "    return mnist_ds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 构建神经网络"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "LeNet网络属于7层神经网络，其中涉及卷积层，全连接层，函数激活等算法，在MindSpore中都已经建成相关算子只需导入使用，如下先将卷积函数，全连接函数，权重等进行初始化，然后在LeNet5中定义神经网络并使用`construct`构建网络。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import mindspore.nn as nn\n",
    "from mindspore.common.initializer import TruncatedNormal\n",
    "\n",
    "\n",
    "def conv(in_channels, out_channels, kernel_size, stride=1, padding=0):\n",
    "    \"\"\"Conv layer weight initial.\"\"\"\n",
    "    weight = weight_variable()\n",
    "    return nn.Conv2d(in_channels, out_channels,\n",
    "                     kernel_size=kernel_size, stride=stride, padding=padding,\n",
    "                     weight_init=weight, has_bias=False, pad_mode=\"valid\")\n",
    "\n",
    "def fc_with_initialize(input_channels, out_channels):\n",
    "    \"\"\"Fc layer weight initial.\"\"\"\n",
    "    weight = weight_variable()\n",
    "    bias = weight_variable()\n",
    "    return nn.Dense(input_channels, out_channels, weight, bias)\n",
    "\n",
    "def weight_variable():\n",
    "    \"\"\"Weight initial.\"\"\"\n",
    "    return TruncatedNormal(0.02)\n",
    "\n",
    "class LeNet5(nn.Cell):\n",
    "    \"\"\"Lenet network structure.\"\"\"\n",
    "    # define the operator required\n",
    "    def __init__(self):\n",
    "        super(LeNet5, self).__init__()\n",
    "        self.conv1 = conv(1, 6, 5)\n",
    "        self.conv2 = conv(6, 16, 5)\n",
    "        self.fc1 = fc_with_initialize(16 * 5 * 5, 120)\n",
    "        self.fc2 = fc_with_initialize(120, 84)\n",
    "        self.fc3 = fc_with_initialize(84, 10)\n",
    "        self.relu = nn.ReLU()\n",
    "        self.max_pool2d = nn.MaxPool2d(kernel_size=2, stride=2)\n",
    "        self.flatten = nn.Flatten()\n",
    "\n",
    "    # use the preceding operators to construct networks\n",
    "    def construct(self, x):\n",
    "        x = self.conv1(x)\n",
    "        x = self.relu(x)\n",
    "        x = self.max_pool2d(x)\n",
    "        x = self.conv2(x)\n",
    "        x = self.relu(x)\n",
    "        x = self.max_pool2d(x)\n",
    "        x = self.flatten(x)\n",
    "        x = self.fc1(x)\n",
    "        x = self.relu(x)\n",
    "        x = self.fc2(x)\n",
    "        x = self.relu(x)\n",
    "        x = self.fc3(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 定义回调函数EvalCallBack"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "实现思想：每隔n个epoch验证一次模型精度，由于在自定义函数中实现，如需了解自定义回调函数的详细用法，请参考[API说明](https://www.mindspore.cn/api/zh-CN/master/api/python/mindspore/mindspore.train.html?highlight=callback#mindspore.train.callback.Callback)。\n",
    "\n",
    "核心实现：回调函数的`epoch_end`内设置验证点，如下：\n",
    "\n",
    "`cur_epoch % eval_per_epoch == 0`：即每`eval_per_epoch`个epoch结束时，验证一次模型精度。\n",
    "\n",
    "- `cur_epoch`：当前训练过程的epoch数值。\n",
    "- `eval_per_epoch`：用户自定义数值，即验证频次。\n",
    "\n",
    "其他参数解释：\n",
    "\n",
    "- `model`：即是MindSpore中的`Model`函数。\n",
    "- `eval_dataset`：验证数据集。\n",
    "- `epoch_per_eval`：记录验证模型的精度和相应的epoch数，其数据形式为`{\"epoch\":[],\"acc\":[]}`。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from mindspore.train.callback import Callback\n",
    "\n",
    "class EvalCallBack(Callback):\n",
    "    def __init__(self, model, eval_dataset, eval_per_epoch, epoch_per_eval):\n",
    "        self.model = model\n",
    "        self.eval_dataset = eval_dataset\n",
    "        self.eval_per_epoch = eval_per_epoch\n",
    "        self.epoch_per_eval = epoch_per_eval\n",
    "        \n",
    "    def epoch_end(self, run_context):\n",
    "        cb_param = run_context.original_args()\n",
    "        cur_epoch = cb_param.cur_epoch_num\n",
    "        if cur_epoch % self.eval_per_epoch == 0:\n",
    "            acc = self.model.eval(self.eval_dataset, dataset_sink_mode=True)\n",
    "            self.epoch_per_eval[\"epoch\"].append(cur_epoch)\n",
    "            self.epoch_per_eval[\"acc\"].append(acc[\"Accuracy\"])\n",
    "            print(acc)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 定义训练网络并执行"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在保存模型的参数`CheckpointConfig`中，需计算好单个epoch中的step数，再根据需要进行验证模型精度的频次对应，\n",
    "本次示例为1875个step/epoch，按照每两个epoch验证一次的思想，这里设置`save_checkpoint_steps=eval_per_epoch*1875`，\n",
    "其中变量`eval_per_epoch`等于2。\n",
    "\n",
    "参数解释：\n",
    "\n",
    "- `train_data_path`：训练数据集地址。\n",
    "- `eval_data_path`：验证数据集地址。\n",
    "- `train_data`：训练数据集。\n",
    "- `eval_data`：验证数据集。\n",
    "- `net_loss`：定义损失函数。\n",
    "- `net-opt`：定义优化器函数。\n",
    "- `config_ck`：定义保存模型信息。\n",
    "    - `save_checkpoint_steps`：每多少个step保存一次模型。\n",
    "    - `keep_checkpoint_max`：设置保存模型数量的上限。\n",
    "- `ckpoint_cb`：定义模型保存的名称及路径信息。\n",
    "- `model`：定义模型。\n",
    "- `model.train`：模型训练函数。\n",
    "- `epoch_per_eval`：定义收集`epoch`数和对应模型精度信息的字典。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch: 1 step: 375, loss is 2.3058078\n",
      "epoch: 1 step: 750, loss is 2.3073978\n",
      "epoch: 1 step: 1125, loss is 2.3103657\n",
      "epoch: 1 step: 1500, loss is 0.65018296\n",
      "epoch: 1 step: 1875, loss is 0.07800862\n",
      "epoch: 2 step: 375, loss is 0.010344766\n",
      "epoch: 2 step: 750, loss is 0.052723818\n",
      "epoch: 2 step: 1125, loss is 0.08183526\n",
      "epoch: 2 step: 1500, loss is 0.007430988\n",
      "epoch: 2 step: 1875, loss is 0.0076965275\n",
      "{'Accuracy': 0.9753605769230769}\n",
      "epoch: 3 step: 375, loss is 0.11964749\n",
      "epoch: 3 step: 750, loss is 0.04522314\n",
      "epoch: 3 step: 1125, loss is 0.018271001\n",
      "epoch: 3 step: 1500, loss is 0.006928641\n",
      "epoch: 3 step: 1875, loss is 0.15374172\n",
      "epoch: 4 step: 375, loss is 0.12120275\n",
      "epoch: 4 step: 750, loss is 0.122824214\n",
      "epoch: 4 step: 1125, loss is 0.0023852547\n",
      "epoch: 4 step: 1500, loss is 0.018273383\n",
      "epoch: 4 step: 1875, loss is 0.08102103\n",
      "{'Accuracy': 0.9821714743589743}\n",
      "epoch: 5 step: 375, loss is 0.12944886\n",
      "epoch: 5 step: 750, loss is 0.0010141768\n",
      "epoch: 5 step: 1125, loss is 0.0054096584\n",
      "epoch: 5 step: 1500, loss is 0.0022614016\n",
      "epoch: 5 step: 1875, loss is 0.07229582\n",
      "epoch: 6 step: 375, loss is 0.0025749032\n",
      "epoch: 6 step: 750, loss is 0.06261393\n",
      "epoch: 6 step: 1125, loss is 0.021273317\n",
      "epoch: 6 step: 1500, loss is 0.011360342\n",
      "epoch: 6 step: 1875, loss is 0.12855275\n",
      "{'Accuracy': 0.9853766025641025}\n",
      "epoch: 7 step: 375, loss is 0.09330422\n",
      "epoch: 7 step: 750, loss is 0.002063415\n",
      "epoch: 7 step: 1125, loss is 0.0047940286\n",
      "epoch: 7 step: 1500, loss is 0.0052507296\n",
      "epoch: 7 step: 1875, loss is 0.018066114\n",
      "epoch: 8 step: 375, loss is 0.08678668\n",
      "epoch: 8 step: 750, loss is 0.02440551\n",
      "epoch: 8 step: 1125, loss is 0.0017507032\n",
      "epoch: 8 step: 1500, loss is 0.02957578\n",
      "epoch: 8 step: 1875, loss is 0.0023948685\n",
      "{'Accuracy': 0.9863782051282052}\n",
      "epoch: 9 step: 375, loss is 0.012376097\n",
      "epoch: 9 step: 750, loss is 0.029711302\n",
      "epoch: 9 step: 1125, loss is 0.017438065\n",
      "epoch: 9 step: 1500, loss is 0.015443239\n",
      "epoch: 9 step: 1875, loss is 0.0031764025\n",
      "epoch: 10 step: 375, loss is 0.0005294987\n",
      "epoch: 10 step: 750, loss is 0.0015696918\n",
      "epoch: 10 step: 1125, loss is 0.019949459\n",
      "epoch: 10 step: 1500, loss is 0.004248183\n",
      "epoch: 10 step: 1875, loss is 0.07389321\n",
      "{'Accuracy': 0.9824719551282052}\n"
     ]
    }
   ],
   "source": [
    "from mindspore.train.callback import ModelCheckpoint, CheckpointConfig, LossMonitor\n",
    "from mindspore.train import Model\n",
    "from mindspore import context\n",
    "from mindspore.nn.metrics import Accuracy\n",
    "from mindspore.nn.loss import SoftmaxCrossEntropyWithLogits\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    context.set_context(mode=context.GRAPH_MODE, device_target=\"GPU\")\n",
    "    train_data_path = \"./MNIST_Data/train\"\n",
    "    eval_data_path = \"./MNIST_Data/test\"\n",
    "    ckpt_save_dir = \"./lenet_ckpt\"\n",
    "    epoch_size = 10\n",
    "    eval_per_epoch = 2\n",
    "    repeat_size = 1\n",
    "    network = LeNet5()\n",
    "    \n",
    "    train_data = create_dataset(train_data_path, repeat_size=repeat_size)\n",
    "    eval_data = create_dataset(eval_data_path, repeat_size=repeat_size)\n",
    "    \n",
    "    # define the loss function\n",
    "    net_loss = SoftmaxCrossEntropyWithLogits(sparse=True, reduction='mean')\n",
    "    # define the optimizer\n",
    "    net_opt = nn.Momentum(network.trainable_params(), learning_rate=0.01, momentum=0.9)\n",
    "    config_ck = CheckpointConfig(save_checkpoint_steps=eval_per_epoch*1875, keep_checkpoint_max=15)\n",
    "    ckpoint_cb = ModelCheckpoint(prefix=\"checkpoint_lenet\", directory=ckpt_save_dir, config=config_ck)\n",
    "    model = Model(network, net_loss, net_opt, metrics={\"Accuracy\": Accuracy()})\n",
    "    \n",
    "    epoch_per_eval = {\"epoch\": [], \"acc\": []}\n",
    "    eval_cb = EvalCallBack(model, eval_data, eval_per_epoch, epoch_per_eval)\n",
    "    \n",
    "    model.train(epoch_size, train_data, callbacks=[ckpoint_cb, LossMonitor(375), eval_cb],\n",
    "                dataset_sink_mode=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在同一目录的文件夹中可以看到`lenet_ckpt`文件夹中，保存了5个模型，和一个计算图相关数据，其结构如下："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "lenet_ckpt\n",
    "├── checkpoint_lenet-10_1875.ckpt\n",
    "├── checkpoint_lenet-2_1875.ckpt\n",
    "├── checkpoint_lenet-4_1875.ckpt\n",
    "├── checkpoint_lenet-6_1875.ckpt\n",
    "├── checkpoint_lenet-8_1875.ckpt\n",
    "└── checkpoint_lenet-graph.meta\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 绘制不同epoch下模型的精度"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "定义绘图函数`eval_show`，将`epoch_per_eval`载入到`eval_show`中，绘制出不同`epoch`下模型的验证精度折线图。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEWCAYAAABxMXBSAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3debzWY/7H8ddb0SLZygyisgxiZEiWkMk+wxhb9m1sQxnrjzBmyJKxm9Ega3ZjnRgVmsrUYDohIhQmEsqWpRHV5/fH9T26O85yn5z7fM/yfj4e53Hu+/4u9+c+y/25r+v6Xp9LEYGZmVmxlso7ADMza1ycOMzMrFacOMzMrFacOMzMrFacOMzMrFacOMzMrFacOKzkJHWRFJJaFrHvEZLG1UdczYWkgyU98QOOHy7p8LqMqYbnK/rvxfLhxGGLkfRfSd9I6lDh8Rezf+Yu+URmSyoi7oqInYvZV9J5ku6scPxuETG0NNHVj+xvd52842gqnDisMm8DB5bfkfRToE1+4TQMjfETcGOMuS4199dfKk4cVpk7gMMK7h8O3F64g6TlJd0uabak6ZJ+L2mpbFsLSZdL+kjSW8AvKzn2ZknvS3pP0oWSWhQTmKT7JX0gaY6kpyVtWLCtjaQrsnjmSBonqU22bRtJ/5b0maR3JR2RPT5G0tEF51isqyz7pNpP0lRgavbYNdk5Ppc0UdK2Bfu3kHS2pDclfZFtX0PSYElXVHgtj0o6uZLXeL2kyys89ndJp2a3BxSc/1VJe1WIf7ykqyR9ApxXyWuqNH5JuwJnA/tL+lLSpIo/I0lLZb/r6ZJmZX8Dy2fbyruYDpf0Tvb7P6ea32WVv6/MwZWdR1JPSc9kv8v3JV0raZmqfmeSns42Tcpe1/5VxWRFigh/+eu7L+C/wI7A68AGQAvgXaAzEECXbL/bgb8DywFdgDeAo7JtvwVeA9YAVgJGZ8e2zLY/AtwALAusAvwHOC7bdgQwrpr4fpM9ZyvgauDFgm2DgTHA6lncW2f7rQl8QWpFLQ2sDGySHTMGOLrgHIs9fxb3k9nraJM9dkh2jpbAacAHQOts2/8BLwPrAQK6Z/v2BGYCS2X7dQDmAj+q5DVul/3Mld1fEfgfsFp2fz9gNdIHv/2Br4BVC+KfD5yYxdemktdUXfznAXdWiOe7n1H2858GrAW0Ax4C7si2dcl+Xjdmz9sdmAdsUMXvsqrfV7XnATYDtszi7wJMAU6u4XcWwDp5/381la/cA/BXw/piUeL4PTAI2DX7J2yZ/fN1yf7J5wHdCo47DhiT3f4n8NuCbTtnx7YEfpQd26Zg+4HA6Oz2Ym9yNcS6Qnbe5bM30f8B3SvZ7yzg4SrO8d2bYmXPn52/Tw1xfFr+vKSEu2cV+00Bdspu9wcer2I/Ae8A22X3jwH+Wc3zv1j+nFn871TYXu3PtEL851F94hgFnFCwbT3g24I38QA6FWz/D3BAJc9Z3e+r6PNk204u/P1W9jvDiaNOv9xVZVW5AziI9KZze4VtHYBlgOkFj00nfXKE9Gn43QrbynUmfep/P+tq+IzU+lilpoCybqBLsm6az0lJrjyeDkBr4M1KDl2jiseLVfhakHSapClZ98pnpMRVfjFBdc81lPRpn+z7HZXtFOmd7l4WjTMdBNxV8PyHKV2sUP7z26jg+b8Xb0U1xF+T1fj+7738A0G5DwpuzyW1TCqq7vdV7Xkk/UTSY1mX5efAxZXEX+3PwH4YJw6rVERMJw2S/4LUHVHoI9KnzM4Fj60JvJfdfp/0Blq4rdy7pBZHh4hYIftqHxEbUrODgD1JLaLlSZ9MIX1C/wj4Gli7kuPereJxSN08bQvu/7iSfb4rIZ2NB5wJ9AVWjIgVgDlZDDU9153AnpK6k7oBH6liP4B7gH0ldQa2AB7Mnr8zqQunP7By9vyTC55/sXgrKiL+msplz+T7v/f5wIc1HFdRdb+vmlxH6gpdNyLak8ZlVGEfl/0uIScOq85RpCb/V4UPRsQC4G/ARZKWy97MTiW9MZJt+52kTpJWBAYUHPs+8ARwhaT22WDr2pJ6FxHPcqSk8zHpzf7igvMuBG4BrpS0WtY62UpSK9Kn9R0l9ZXUUtLKkjbJDn0R2FtSW6XLNY8qIob5wGygpaQ/AO0Ltt8EXCBpXSUbS1o5i3EGMIHU0ngwIv5X1ZNExAvZc9wEjIyIz7JNy5LeFGcDSDqS1OIoVk3xfwh0UXahQyXuAU6R1FVSO9Lv4L6ImF+LGGr6fRXzGj4HvpS0PnB8Ecd8SBqXsTrgxGFViog3I6Ksis0nkj6tvwWMA+4mvRFA+kQ8EpgEPM/3WyyHkbq6XiX1rz8ArFpESLeTukbey459tsL200kD0xOAT4A/kQaj3yG1nE7LHn+RNOAKcBXwDemNZSgFXUJVGAkMJ10MMJ30qbmwW+RKUuJ8gvTmdjOLX8o8FPgpVXRTVXAPqXV1d/kDEfEqcAXwTBbzT4HxRZyr2Pjvz75/LOn5So6/JYv9aVKL9GvS38KSqPT3VeRxB5EueLgRuK+IY84Dhmbde32XKFr7TvlVG2ZWDyRtR2qZdck+dZs1Om5xmNUTSUsDJwE3OWlYY+bEYVYPJG0AfEbqkrs653DMfhB3VZmZWa24xWFmZrXSLAqAdejQIbp06ZJ3GGZmjcrEiRM/ioiOFR9vFomjS5culJVVdVWpmZlVRtL0yh53V5WZmdWKE4eZmdWKE4eZmdWKE4eZmdWKE4eZmdWKE4eZmdWKE4eZmdVKs5jHYWaN3MyZMG4cfPUVHHYYtGiRd0TNmhOHmTUsEfD66ylR/Otf6ftbby3a/thjcNdd0Lp1fjE2c04cZpavb7+FF15YPFF89FHa1rEjbLMN9O+fvo8bB6eeCr/8JTzyCCy3XL6xN1NOHGZWv778Ep59dlGiePZZmDs3bVt77ZQUtt02JYqf/ARUsJz45ptDhw5w5JHQpw88/nhKLlavnDjMrLQ+/BDGj1/UmnjhBViwAJZaCrp3h6OPTkmiVy9YbbWaz3foobDiirDffinBPPEErLlm6V+HfaekiUPSrsA1QAvSqmeXVNjembSGcUfSmsOHRMSMbNulwC9JV349CZwUESFpGeBaYHtgIXBORDxYytdhZkWKgDffXLzb6Y030rbWrWGLLeCss1Ki2GoraN9+yZ5n991Twthjj5RwnngCNtig7l6HVatkiUNSC2AwsBMwA5ggaVhEvFqw2+XA7RExVFIfYBBwqKStgV7Axtl+44DewBjgHGBWRPxE0lLASqV6DWZWg/nz4aWXFiWJcePggw/StpVWSm/q5S2KzTaDZZapu+fedlsYOxZ22SXdfvxx6Nmz7s5vVSpli6MnMC0i3gKQdC+wJ1CYOLoBp2S3RwOPZLcDaA0sAwhYGvgw2/YbYH2AbN3mj0r3EsxsMXPnwn/+syhRPPMMfPFF2ta5M+y4Y0oS224L66+fuqNKqXv31A22005pzOORR1IMVlKlTByrA+8W3J8BbFFhn0nAPqTurL2A5SStHBHPSBoNvE9KHNdGxBRJK2THXSBpe+BNoH9EfFjhvEg6FjgWYE33f5otmY8/Xnx8YuLEdBWUBD/9aRpv2Gab9LXGGvnEuPbaKcZddoFf/ALuvhv23TefWJqJUiYOVfJYxQXOTweulXQE8DTwHjBf0jrABkCnbL8nJW1Haq10AsZHxKmSTiV1dx36vSeKGAIMAejRo4cXVjerSQRMn74oSfzrXzBlStq2zDKpG+i001JrYqut0gB1Q7Hqqqnbao89oG9fuO46OO64vKNqskqZOGYAhR9BOgEzC3eIiJnA3gCS2gH7RMScrLXwbER8mW0bDmwJ/AuYCzycneJ+4KgSvgazpmvBApg8edHYxL/+Be+9l7Ytv3wanzj00JQoevRo+BPuVlwxDZLvtx/89reptXTWWYtfzmt1opSJYwKwrqSupJbEAcBBhTtI6gB8ko1VnEW6wgrgHeAYSYNILZfewNXZVVWPkq6o+iewA4uPmZhZVb7+GiZMWJQk/v1vmDMnbVt99ZQgyudPbLhh4yzr0bZtGuc48kg455w0kfDyy0s/1tLMlCxxRMR8Sf2BkaTLcW+JiFckDQTKImIYKQEMkhSkrqp+2eEPAH2Al0ndWyMi4tFs25nAHZKuBmYDR5bqNZg1ap99lvr+yxPFhAnwzTdpW7dusP/+ixJF585N55P50kvD7benq7quuiq1PG66KT1udUIRTb/7v0ePHlFWVpZ3GGal9e67i8+fmDw5jVu0bJm6msqvdtp66zT7uqmLgAsvhD/8IY193HcftGmTd1SNiqSJEdGj4uOeOW7WGC1cmAauCxPF9OlpW7t2KTn07ZuSRc+eqQunuZHg3HNTkuzXL1119eijafzGfhAnDrPG4Jtv0qWw5Yli/Hj45JO07cc/Tgni1FPT9403Tq0MS44/PnVbHXoo9O4NI0akn5ktMf91mTVEn3+eJteVtyaeey4NbkMq/LfXXou6ntZaq+mMT5TK/vvDCivA3nunn9uTT0LXrnlH1Wg5cZg1BO+/v3i306RJqTuqRQv42c/Sp+byiXarrJJ3tI3TLrvAqFFpkmCvXjByZJrEaLXmxGGWl4kT4S9/SYnizTfTY23bpsl1556bksSWW6YxC6sbW26ZkvPOO8N228E//pHGg6xWnDjM6lsEXHttmoXdrh1svz2ccELqdtpkE182WmobbpjGiHbeOdW1euCB1AqxojlxmNWnOXNStdgHHkiXiN52Wxq4tfrVpUtq6e26K+y5JwwdCgcdVONhlng6pVl9eeGFVFr84Yfhssvg73930sjTKqvAmDFpvOPgg1O3oRXFicOs1CLghhvS2MXXX6difKef7iuhGoL27dPlub/+Nfzud/DHP6bfl1XLicOslL78Eg45JBXd23771Oro1SvvqKxQ69Zw//2pvtXAgdC/f7qizarkMQ6zUnn55VSpdepUuOgiGDDAxfYaqpYt4eab0yzzyy5LkyuHDq3bFQubECcOs1K49dZU5mL55dPcge23zzsiq4kEl16akseZZ8Knn8KDD8Kyy+YdWYPjjz9mdWnu3NTl8ZvfpPkBL77opNHYnHFGqqb75JNpSdry0i72HScOs7oyZUoqKDh0aBpkHTkSfvSjvKOyJXHUUemS6YkT00TB8gWuDHDiMKsbd94Jm28Os2alhHHeeY1zISRbZK+90hVX06enWfxTp+YdUYPhxGH2Q/zvf3Dssany6mabpa6pnXbKOyqrKz//OYwena6O22abdFWcOXGYLbGpU9PcjBtvTGtbjxoFq62Wd1RW13r0SLPMW7VK41Vjx+YdUe6cOMyWxN/+lloY776bCuVdfLHXwGjK1lsv1bdaffVUZXfYsLwjypUTh1ltzJuXJojtvz9stFHqmnKBvOZhjTVSZd3u3dO6HrfdlndEuXHiMCvW22+nWd+DB6fKtmPHpjcTaz5WXjl1Sfbpky67vuKKvCPKhROHWTEeeSQtqPTmm+n25Ze7/Hlz1a5dWrt8v/1SzbGzzmp29a3cKWtWnW++SaVCrroqXW57331ectTSQPk996TqxpdcAh99BNdf32wuwXbiMKvKO+9A375pve8TT0w1jFq1yjsqayhatIDrroOOHeHCC9MM87vuSkUTmzgnDrPK/OMfcNhh8O23qXLqvvvmHZE1RBJccEEa+zjlFPjlL1NX5nLL5R1ZSXmMw6zQ/Pmpa2r33WHNNeH55500rGYnnwy3354umOjTB2bPzjuiknLiMCv33ntppvCf/gTHHQfPPAPrrJN3VNZYHHpoam1MnpzWj3/nnbwjKhknDjOAJ56ATTZJJSXuvjsNdDaDvmqrY7vvnv6WPvggXbo9ZUreEZVESROHpF0lvS5pmqQBlWzvLGmUpJckjZHUqWDbpZJekTRF0p+lxdfZlDRM0uRSxm/NwIIF8Ic/wK67wo9/DGVlcOCBeUdljdm226Yuq2+/TbcnTMg7ojpXssQhqQUwGNgN6AYcKKlbhd0uB26PiI2BgcCg7NitgV7AxsBGwOZA74Jz7w18WarYrZn44INUkPCCC+CII9LVU+uvn3dU1hR0755KlLRvn7o/n3oq74jqVClbHD2BaRHxVkR8A9wL7Flhn27AqOz26ILtAbQGlgFaAUsDHwJIagecClxYwtitqRs9OnVNPftsKh1xyy3Qtm3eUVlTsvbaKXmstVa62uqBB/KOqM6UMnGsDrxbcH9G9lihScA+2e29gOUkrRwRz5ASyfvZ18iIKO8svAC4Aphb3ZNLOlZSmaSy2U38CgerhYUL0zX3O+4IK64I//kPHH543lFZU7XqqqnbavPN05ygIUPyjqhOlDJxqJLHKs7LPx3oLekFUlfUe8B8SesAGwCdSMmmj6TtJG0CrBMRD9f05BExJCJ6RESPjh07/qAXYk3E7Nmw225w7rlpHGPChFSo0KyUVlwxDZjvtlu6Wm/QoEZfoqSUEwBnAIUV4DoBMwt3iIiZwN7wXRfUPhExR9KxwLMR8WW2bTiwJfAFsJmk/2axryJpTERsX8LXYU3BuHFwwAGpNMSQIXD00Wnylll9aNs2Xap75JFw9tnpQ8zll8NSjfPC1lJGPQFYV1JXScsABwCLFbGX1EFSeQxnAbdkt98htURaSlqa1BqZEhHXRcRqEdEF2AZ4w0nDqrVwYZqXsf320KZNGtM45hgnDat/Sy+dJgmeeGKqfXbkkenKq0aoZC2OiJgvqT8wEmgB3BIRr0gaCJRFxDBge2CQpACeBvplhz8A9AFeJnVvjYiIR0sVqzVRH3+cxi/+8Y/Uv3zjjekqF7O8LLUUXHNNqm/1hz/Ap5+mwplt2uQdWa0oGnlfWzF69OgRZWVleYdh9enZZ9NiSx98kD7dHX+8WxnWsFx3HfTrl9Yyf/RRWH75vCP6HkkTI6JHxccbZwebWVUiUqLYdttUvXT8eDjhBCcNa3iOPz6VZn/2WejdO33IaSScOKzp+Owz2GcfOPXUVPrh+eehx/c+LJk1HPvvn1obU6emlsfbb+cdUVGcOKxpmDgRNt00/RNedRU89BCssELeUZnVbJdd0nK0n3yS6lu9/HLeEdXIicMat4i0BvjWW6eS6P/6Vypx7a4pa0y23DL97Uqw3Xbw73/nHVG1nDis8fr88zQ3o3//VHPqhRfSP6BZY7ThhmlMrmPHVNlg+PC8I6qSE4c1TpMmpfGLBx9M8zSGDUursJk1Zl26pMmq668Pv/pVGjxvgJw4rHGJSPMxttgCvvoKxoyBM85otDNwzb5nlVXS33WvXnDwwXDttXlH9D3+b7PG48sv0zrgxx6bLl984YV0JYpZU9O+PYwYAXvumWaan3deg6pv5cRhjcMrr6QKo3ffndbPGD48fTIza6pat4b770+lSc4/PyWQhQvzjgoobZFDs7oxdGiaLNW+fVoQ5+c/zzsis/rRsiXcfDN06ACXXZYu2b3tNlhmmXzDyvXZzaozd276lHXLLalI4T33pOVdzZoTCS69NCWPM89M9a0eeACWXTa3kNxVZQ3T66+nAfBbb03rZzz1lJOGNW9nnAE33ZTW9thpp9T6yIkThzU8d98Nm22WaveMGAEDB6a6U2bN3VFHpdbGxIlpouB77+USRo2JI1t+tZ+kFesjIGvGvv4afvvbdAniz34GL74IO++cd1RmDctee6UPVNOnp6sKp06t9xCKaXEcAKwGTJB0r6RdJNdzsDo2bRpstRXccEPqxx09GlavuES9mQHpApHRo9Ml6ttsky5Nr0c1Jo6ImBYR5wA/Ae4mrdL3jqTzJa1U6gCtGXjwwdQ1NX16KlJ4ySXpahIzq1qPHmmWeatW6eKRsWPr7amLGuOQtDFwBXAZ8CCwL/A58M/ShWZN3rx58Lvfwb77QrduqWtq993zjsqs8VhvvVTfavXVU5XdYcNqPqYOFDPGMRG4irSG+MYR8buIeC4irgDeKnWA1kS9/XZabOkvf4FTTkmfltZcM++ozBqfNdZIlXW7d4e9907znkqsmP6A/SKi0gQREXvXcTzWHAwbltYCj0jrZuy1V94RmTVuK6+c1vTYe2844gj4+OO0oFmJFNNVdbSk71bEkbSipAtLFpE1Xd9+C6efnurvrL12WqHPScOsbrRrl8YI99sPTjsNzj67ZPWtikkcu0XEZ+V3IuJT4BclicaarnffTYUJr7gC+vVL/bJrrZV3VGZNS6tWqcLCccfBoEHp+4IFdf40xSSOFpJald+R1AZoVc3+ZosbPjzNy5g8Ge67L5WJbuU/IbOSaNECrrsOfv/7VHnh+efr/CmKSRx3AqMkHSXpN8CTQOlHX6zxmz8/NZd/8Yt01UdZGfTtm3dUZk2flKpIT56cqkrXsRoHxyPiUkkvAzsAAi6IiJF1Hok1LTNnwoEHwtNPwzHHwDXXQJs2eUdl1ryst15JTlvULKuIGA403AVwrWF56ik46KC0Qt8dd8Ahh+QdkZnVoWLmcWwpaYKkLyV9I2mBpM/rIzhrZBYsSCuV7bxzWmSprMxJw6wJKmaM41rgQGAq0AY4GvhLKYOyRuqEE9JKZYcdBs89BxtskHdEZlYCRZUciYhpQIuIWBARtwJFLcEmaVdJr0uaJmlAJds7Sxol6SVJYyR1Kth2qaRXJE2R9GclbSX9Q9Jr2bZLin2hVmJjx8KQIWnS0W235brIjJmVVjGJY66kZYAXszfzU4Aa3xUktQAGA7sB3YADJXWrsNvlwO0RsTEwEBiUHbs10AvYGNgI2BzoXX5MRKwP/AzoJWm3Il6DldK8eakceteu6UoOM2vSikkch2b79Qe+AtYA9iniuJ7AtIh4KyK+Ae4F9qywTzdgVHZ7dMH2AFoDy5DmjCwNfBgRcyNiNEB2zueBTli+Lr0UXnsNBg+Gtm3zjsbMSqzaxJG1Gi6KiK8j4vOIOD8iTs26rmqyOvBuwf0Z2WOFJrEoCe0FLCdp5Yh4hpRI3s++RkbElAqxrQDswaLEUzH2Y7NFqMpmz55dRLi2RKZOhYsuSvMzdnPjz6w5qDZxRMQCoGPWVVVblS32VLFwyulAb0kvkLqi3gPmS1oH2IDUmlgd6CNpu+9OLLUE7gH+XE0BxiER0SMienTs2HEJwrcaRaQB8Vat4Oqr847GzOpJMfM4/guMlzSM1FUFQERcWcNxM0jdWuU6ATMLd4iImcDeAJLaAftExBxJxwLPRsSX2bbhwJbA09mhQ4CpEeF3qzzdfXeaszF4MKy6at7RmFk9KWaMYybwWLbvcgVfNZkArCupa9ZiOQBYbJURSR0klcdwFml1QYB3SC2RlpKWJrVGpmTHXAgsD5xcRAxWKp98kq6g6tkzFVIzs2ajmJIj5y/JiSNivqT+wEigBXBLRLwiaSBQFhHDgO2BQZKC1Jrolx3+ANAHeJnUvTUiIh7NLtc9B3gNeD5b+vzaiLhpSWK0H2DAgFTzf+TIVFTNzJoNRQ312iWN5vtjE0REn1IFVdd69OgRZWVleYfRdIwfD9tsk2r+X3553tGYWYlImhgRPSo+XswYx+kFt1uTroKaX1eBWSPz7bdpzsYaa6TyImbW7BTTVTWxwkPjJY0tUTzW0F1xRSrVPGxYWnHMzJqdGhOHpJUK7i4FbAb8uGQRWcP11lswcGBa7nWPPfKOxsxyUkxX1UTSGIdIXVRvA0eVMihrgCLSkq8tWsCf/5x3NGaWo2K6qrrWRyDWwN1/P4wYkSb6dXKVF7PmrJj1OPpl5T3K768o6YTShmUNypw5cNJJsOmm0L9/3tGYWc6KmQB4TER8Vn4nIj4FjildSNbgnH02zJqVyqZ7zoZZs1dM4lhK2Uw7+K7w4ZLUrrLG6Lnn4LrrUktjs83yjsbMGoBiBsdHAn+TdD1pkPy3wIiSRmUNw/z5qZzIqqt6nQ0z+04xieNM4FjgeNKVVU8ALvHRHFxzDUyaBA8+CO3b5x2NmTUQxSSONsCNEXE9fNdV1QqYW8rALGfTp8Mf/gC7757mbZiZZYoZ4xhFSh7l2gBPlSYcaxAi4MQT0+1rrwVVtrSKmTVXxbQ4WpeviwEQEV9K8vqgTdkjj8Cjj8Jll0HnznlHY2YNTDEtjq8kbVp+R9JmwP9KF5Ll6osvUmtj443T3A0zswqKaXGcDNwvqXz1vlWB/UsXkuXq3HNh5kx44AFYeum8ozGzBqiYkiMTJK0PrEe6quq1iPi25JFZ/Zs4Ef7yl1Q2fcst847GzBqoYlockJJGN9J6HD+TRETcXrqwrN4tWJDmbKyyClx8cd7RmFkDVkxZ9T+SlnjtBjwO7AaMA5w4mpLBg1OL4957YYUVat7fzJqtYgbH9wV2AD6IiCOB7qR5HNZUzJgB55wDu+wCffvmHY2ZNXDFJI7/RcRCYL6k9sAsYK3ShmX16qSTUnmRv/7VczbMrEbFjHGUZWXVbyQt6vQl8J+SRmX159FH4aGH0rjGWv48YGY1U0QUv7PUBWgfES+VKqBS6NGjR5SVleUdRsPz1VfQrRsstxw8/zws46LHZraIpIkR0aPi48VeVQVARPy3ziKy/J13HrzzDowb56RhZkUrZozDmqJJk+Cqq+Doo6FXr7yjMbNGxImjOSqfs7HSSvCnP+UdjZk1MlV2VUlaqboDI+KTug/H6sWQIWllvzvuSMnDzKwWqhvjmEha8a+y6zMDX5LbOL3/PgwYADvsAAcfnHc0ZtYIVdlVFRFdI2Kt7HvFr6KShqRdJb0uaZqkAZVs7yxplKSXJI2R1Klg26WSXpE0RdKfy9c9l7SZpJezc373uBXplFNg3ry0jrh/dGa2BGoc41ByiKRzs/trSupZxHEtgMGkEiXdgAMldauw2+XA7RGxMTAQGJQduzXQC9gY2AjYHOidHXMdaSnbdbOvXWuKxTIjRsB998HZZ8O66+YdjZk1UsUMjv8V2Ao4KLv/BSkh1KQnMC0i3oqIb4B7gT0r7NONtMIgwOiC7UEqqLgMqbzJ0sCHklYlzSN5JtIElNuBXxcRi82dCyecAOutB2eemXc0ZtaIFZM4toiIfsDXABHxKekNvSarA+8W3J+RPVZoErBPdnsvYDlJK0fEM6RE8n72NTIipmTHzyDc73oAABM2SURBVKjhnABIOlZSmaSy2bNnFxFuE3fhhfD223D99dDKpcbMbMkVkzi+zbqdAkBSR2BhEcdVNahe6HSgt6QXSF1R75FqYq0DbAB0IiWGPpK2K/Kc6cGIIRHRIyJ6dOzYsYhwm7DJk9MysEccAdtvn3c0ZtbIFZM4/gw8DKwi6SJSSfViFmyYAaxRcL8TMLNwh4iYGRF7R8TPgHOyx+aQWh/PRsSX2Xrnw4Ets3N2qu6cVsHChWlhpuWXT8nDzOwHqjFxRMRdwBmkgev3gV9HxP1FnHsCsK6krpKWAQ4AhhXuIKmDpPIYzgJuyW6/Q2qJtJS0NKk1MiUi3ge+kLRldjXVYcDfi4il+br5Zhg/PiWNDh3yjsbMmoBiJwDOAu4p3FbTBMCImC+pPzASaAHcEhGvSBoIlEXEMNICUYMkBfA00C87/AGgD/AyqStqREQ8mm07HrgNaENqiQwv7qU2Q7NmpYHw7bZL3VRmZnWgyuq4kt5m0QTANYFPs9srAO9ERNf6CvKHarbVcQ89NF1++9JLsP76eUdjZo1MVdVxa5wASGox7BERHSJiZWB34KHShWp14qmn4M470yxxJw0zq0PFDI5vHhGPl9+JiOEsmoxnDdHXX8Pxx8M666TJfmZmdaiY9Tg+kvR74E5S19UhwMcljcp+mIsvhmnT4MknoXXrvKMxsyammBbHgUBH0iW5jwCrZI9ZQ/Taa3DJJamA4Y475h2NmTVBNbY4squnTpLUHliYzauwhigizdlYdlm44oq8ozGzJqqYIoc/zWZ2vwy8ImmipI1KH5rV2tChMHYsXHop/OhHeUdjZk1UMV1VNwCnRkTniOgMnAYMKW1YVmsffQSnn56WgT3qqLyjMbMmrJjEsWxEjC6/ExFjgGVLFpEtmf/7P5gzJxUxXMorAptZ6RRzVdVb2Vocd2T3DwHeLl1IVmtjx8Jtt6U5Gxu5F9HMSquYj6a/IV1V9RDpyqqOwJGlDMpqYd68NCDetSuce27e0ZhZM1DMVVWfAr+rh1hsSVx6aboE9/HHoW3bvKMxs2aguiKHw6raBhARv6r7cKxWpk6Fiy6Cvn1ht93yjsbMmonqWhxbkVbwuwd4jsoXUbK8RKSyIq1awdVX5x2NmTUj1SWOHwM7kWaJHwT8A7gnIl6pj8CsBnffDaNGweDBsOqqeUdjZs1IddVxF0TEiIg4nLT63jRgjKQT6y06q9wnn8Cpp0LPnnDccXlHY2bNTLWD45JaAb8ktTq6kJaRdUn1vA0YAB9/DCNHQosWeUdjZs1MdYPjQ4GNSCvsnR8Rk+stKqva+PFw441w2mmwySZ5R2NmzVB1KwAuBL7K7hbuJCAion2JY6szTWYFwG++gU03hc8/h1dfhXbt8o7IzJqwqlYArLLFERGuW9HQXHklvPIKDBvmpGFmuXFyaCzeegsGDoS99oI99sg7GjNrxpw4GoMI6NcvDYT/+c95R2NmzVwxRQ4tb/ffDyNGpIl+nTrlHY2ZNXNucTR0n30GJ52UBsX79887GjMztzgavHPOgVmz4LHHPGfDzBoEtzgasueeg+uuSy2NzTbLOxozM8CJo+GaPz+VE1ltNbjggryjMTP7TkkTh6RdJb0uaZqkAZVs7yxplKSXJI2R1Cl7/OeSXiz4+lrSr7NtO0h6Pnt8nKR1SvkacnPNNTBpUrqKqn2jmWtpZs1AlTPHf/CJpRbAG6QKuzOACcCBEfFqwT73A49FxFBJfYAjI+LQCudZiVRgsVNEzJX0BrBnREyRdALQMyKOqC6WRjdzfPp06NYN+vRJk/3kivZmVv+qmjleyhZHT2BaRLwVEd8A9wJ7VtinGzAquz26ku0A+wLDI2Judj+A8o/gywMz6zTqvEXAiVkB4muvddIwswanlIljddJCUOVmZI8VmgTsk93eC1hO0soV9jmAtJhUuaOBxyXNAA4FLqmziBuCRx6BRx+F88+Hzp3zjsbM7HtKmTgq+6hcsV/sdKC3pBeA3sB7wPzvTiCtCvwUGFlwzCnALyKiE3ArcGWlTy4dK6lMUtns2bOX/FXUpy++SK2N7t3T3A0zswaolPM4ZgBrFNzvRIVupYiYCewNIKkdsE9EzCnYpS/wcER8m+3TEegeEc9l2+8DRlT25BExBBgCaYzjB7+a+nDuuTBzJjz4ICy9dN7RmJlVqpQtjgnAupK6SlqG1OU0rHAHSR0klcdwFnBLhXMcyOLdVJ8Cy0v6SXZ/J2BKnUeeh4kT4S9/SeuIb7FF3tGYmVWpZC2OiJgvqT+pm6kFcEtEvCJpIFAWEcOA7YFBkgJ4GuhXfrykLqQWy9gK5zwGeDBbL+RT4Deleg31ZsGCNGdjlVXg4ovzjsbMrFolLTkSEY8Dj1d47A8Ftx8AHqji2P/y/cF0IuJh4OE6DTRvgwenFse998Lyy+cdjZlZtTxzPG8zZqR6VLvuCn375h2NmVmNnDjydtJJqbzI4MGes2FmjYKr4+bp0UfhoYdg0CBYa628ozEzK4pbHHn56qtU9XbDDeG00/KOxsysaG5x5OW88+Cdd2DcOM/ZMLNGxS2OPEyaBFddBcccA7165R2NmVmtOHHUt/I5GyutBJc0rTJbZtY8uKuqvt1wQ1rZ7847U/IwM2tk3OKoT++/D2edBTvuCAcdlHc0ZmZLxImjPp1yCsybB3/9q+dsmFmj5cRRX0aMgPvuS7PE110372jMzJaYE0d9mDsXTjgB1lsPzjgj72jMzH4QD47XhwsugLffhjFjoFWrvKMxM/tB3OIotcmT4fLL4YgjoHfvvKMxM/vBnDhKaeFC+O1vU6n0yy7LOxozszrhrqpSuvlmGD8ebr0VOnTIOxozszrhFkepfPhhGgjv3RsOPzzvaMzM6owTR6mcdlqqgHv99Z6zYWZNihNHKTz1FNx1FwwYAOuvn3c0ZmZ1yomjrn39NRx/PKyzDpx9dt7RmJnVOQ+O17WLL4Zp0+DJJ6F167yjMTOrc25x1KXXXkul0g8+OBUyNDNrgpw46kpEmrPRrh1ceWXe0ZiZlYy7qurK0KEwdiwMGQKrrJJ3NGZmJeMWR1346CM4/fS0DOxRR+UdjZlZSTlx1IX/+z+YMyfN2VjKP1Iza9r8LvdDjRkDt92WWhwbbZR3NGZmJVfSxCFpV0mvS5omaUAl2ztLGiXpJUljJHXKHv+5pBcLvr6W9OtsmyRdJOkNSVMk/a6Ur6Fa8+alAfGuXeHcc3MLw8ysPpVscFxSC2AwsBMwA5ggaVhEvFqw2+XA7RExVFIfYBBwaESMBjbJzrMSMA14IjvmCGANYP2IWCgpv5HoSy+F11+H4cOhbdvcwjAzq0+lbHH0BKZFxFsR8Q1wL7BnhX26AaOy26Mr2Q6wLzA8IuZm948HBkbEQoCImFXnkRdj6lS46CLo2xd23TWXEMzM8lDKxLE68G7B/RnZY4UmAftkt/cClpO0coV9DgDuKbi/NrC/pDJJwyVVuoC3pGOzfcpmz569xC+iUhGprEirVnD11XV7bjOzBq6UiaOykrBR4f7pQG9JLwC9gfeA+d+dQFoV+CkwsuCYVsDXEdEDuBG4pbInj4ghEdEjInp07NhxyV9FZe6+G0aNgkGDYNVV6/bcZmYNXCknAM4gjUWU6wTMLNwhImYCewNIagfsExFzCnbpCzwcEd9WOO+D2e2HgVvrOO7qffIJnHIKbLEFHHdcvT61mVlDUMoWxwRgXUldJS1D6nIaVriDpA6SymM4i++3Hg5k8W4qgEeAPtnt3sAbdRp1TQYMSMnjhhugRYt6fWozs4agZIkjIuYD/UndTFOAv0XEK5IGSvpVttv2wOuS3gB+BFxUfrykLqQWy9gKp74E2EfSy6SrsI4u1Wv4nvHj4cYb4eSToXv3entaM7OGRBEVhx2anh49ekRZWdkPO8k338Cmm8IXX8Arr6RihmZmTZikidl48mJc5LBYV16ZEsawYU4aZtasueRIMd56C84/H/beG/bYI+9ozMxy5cRRkwjo1w9atoRrrsk7GjOz3Lmrqib33w8jRqSJfp065R2NmVnu3OKozmefwUknwWabQf/+eUdjZtYguMVRnXPOgVmz4LHHPGfDzCzjFkd1unaFM85ILQ4zMwPc4qje6afnHYGZWYPjFoeZmdWKE4eZmdWKE4eZmdWKE4eZmdWKE4eZmdWKE4eZmdWKE4eZmdWKE4eZmdVKs1jISdJsYPoSHt4B+KgOw6krjqt2HFftOK7aaapxdY6IjhUfbBaJ44eQVFbZClh5c1y147hqx3HVTnOLy11VZmZWK04cZmZWK04cNRuSdwBVcFy147hqx3HVTrOKy2McZmZWK25xmJlZrThxmJlZrThxVEHSGpJGS5oi6RVJJ+UdE4Ck1pL+I2lSFtf5ecdUTlILSS9IeizvWApJ+q+klyW9KKks73jKSVpB0gOSXsv+zrZqADGtl/2cyr8+l3Ry3nEBSDol+5ufLOkeSa3zjglA0klZTK/k+bOSdIukWZImFzy2kqQnJU3Nvq9YF8/lxFG1+cBpEbEBsCXQT1K3nGMCmAf0iYjuwCbArpK2zDmmcicBU/IOogo/j4hNGti19tcAIyJifaA7DeBnFxGvZz+nTYDNgLnAwzmHhaTVgd8BPSJiI6AFcEC+UYGkjYBjgJ6k3+HuktbNKZzbgF0rPDYAGBUR6wKjsvs/mBNHFSLi/Yh4Prv9BemfevV8o4JIvszuLp195X6Fg6ROwC+Bm/KOpTGQ1B7YDrgZICK+iYjP8o3qe3YA3oyIJa26UNdaAm0ktQTaAjNzjgdgA+DZiJgbEfOBscBeeQQSEU8Dn1R4eE9gaHZ7KPDrunguJ44iSOoC/Ax4Lt9IkqxL6EVgFvBkRDSEuK4GzgAW5h1IJQJ4QtJEScfmHUxmLWA2cGvWvXeTpGXzDqqCA4B78g4CICLeAy4H3gHeB+ZExBP5RgXAZGA7SStLagv8Algj55gK/Sgi3of0YRhYpS5O6sRRA0ntgAeBkyPi87zjAYiIBVlXQiegZ9Zczo2k3YFZETExzziq0SsiNgV2I3U5bpd3QKRPz5sC10XEz4CvqKNuhLogaRngV8D9eccCkPXN7wl0BVYDlpV0SL5RQURMAf4EPAmMACaRurmbNCeOakhampQ07oqIh/KOp6Ksa2MM3+/XrG+9gF9J+i9wL9BH0p35hrRIRMzMvs8i9df3zDciAGYAMwpaiw+QEklDsRvwfER8mHcgmR2BtyNidkR8CzwEbJ1zTABExM0RsWlEbEfqKpqad0wFPpS0KkD2fVZdnNSJowqSROp/nhIRV+YdTzlJHSWtkN1uQ/qHei3PmCLirIjoFBFdSN0b/4yI3D8NAkhaVtJy5beBnUndC7mKiA+AdyWtlz20A/BqjiFVdCANpJsq8w6wpaS22f/mDjSAiwkAJK2SfV8T2JuG9XMbBhye3T4c+HtdnLRlXZykieoFHAq8nI0nAJwdEY/nGBPAqsBQSS1Iif9vEdGgLn9tYH4EPJzea2gJ3B0RI/IN6TsnAndl3UJvAUfmHA8AWV/9TsBxecdSLiKek/QA8DypK+gFGk6ZjwclrQx8C/SLiE/zCELSPcD2QAdJM4A/ApcAf5N0FCn57lcnz+WSI2ZmVhvuqjIzs1px4jAzs1px4jAzs1px4jAzs1px4jAzs1px4jCrQ5K2z7M6sKQjJF2b1/Nb8+DEYWbfyeYHmVXLicOaHUmHZGuavCjphvI3S0lfSrpC0vOSRknqmD2+iaRnJb0k6eHyNQ0krSPpqWxtlOclrZ09RbuCdTbuymY6V4xhjKQ/ZXG8IWnb7PHFWgySHpO0fUF8f8qKNT4lqWd2nrck/arg9GtIGiHpdUl/LPJ1D5T0HJD7miDW8DlxWLMiaQNgf1Lhw02ABcDB2eZlSfWZNiWVxy5/070dODMiNgZeLnj8LmBwtjbK1qSqrZAqKZ8MdCNVwe1VRTgtI6Jntu8fq9in0LLAmIjYDPgCuJA0w3svYGDBfj2z17QJsJ+kHkW87skRsUVEjCsiDmvmXHLEmpsdSAsUTcgaAm1YVPhtIXBfdvtO4CFJywMrRMTY7PGhwP1Z/avVI+JhgIj4GiA7538iYkZ2/0WgC1DZG3J54cyJ2T41+YZUgRVSApsXEd9KernC8U9GxMfZ8z8EbEMq01HV615AKuZpVhQnDmtuBAyNiLOK2Le6ejzf634qMK/g9gKq/j+bV8k+81m8J6BwedRvY1GNoIXlx0fEwmxxo6riDqp/3V9HxIIqYjT7HndVWXMzCti3oKLpSpI6Z9uWAvbNbh8EjIuIOcCn5WMQpMKXY7O1WWZI+nV2nlZZccAf6r/AJpKWkrQGS1YCfqfsdbUhrfg2nupft1mtuMVhzUpEvCrp96QVAZciq2gKTCctprShpInAHNKYAKRy1NdniaGwiu2hwA2SBmbnqYvKo+OBt0ldUZNJ1WBraxxwB7AOqRpwGUA1r9usVlwd1ywj6cuIaJd3HGYNnbuqzMysVtziMDOzWnGLw8zMasWJw8zMasWJw8zMasWJw8zMasWJw8zMauX/AWFVESTQvY98AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "def eval_show(epoch_per_eval):\n",
    "    plt.xlabel(\"epoch number\")\n",
    "    plt.ylabel(\"Model accuracy\")\n",
    "    plt.title(\"Model accuracy variation chart\")\n",
    "    plt.plot(epoch_per_eval[\"epoch\"], epoch_per_eval[\"acc\"], \"red\")\n",
    "    plt.show()\n",
    "    \n",
    "eval_show(epoch_per_eval)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "从上图可以一目了然地挑选出需要的最优模型。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 总结"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "本例使用MNIST数据集通过卷积神经网络LeNet5进行训练，着重介绍了利用回调函数在进行模型训练的同时进行模型的验证，保存对应`epoch`的模型，并从中挑选出最优模型的方法。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}