From c262a6e388a0da599935c22eb0228863d7860eb4 Mon Sep 17 00:00:00 2001 From: lvmingfu <630944715@qq.com> Date: Wed, 26 Aug 2020 10:17:45 +0800 Subject: [PATCH] add synchronization_training_and_eval files in tutorials --- tutorials/notebook/README.md | 1 + ...chronization_training_and_evaluation.ipynb | 510 ++++++++++++++++++ ...ynchronization_training_and_evaluation.png | Bin 0 -> 12868 bytes ...synchronization_training_and_evaluation.md | 174 ++++++ tutorials/source_zh_cn/index.rst | 1 + 5 files changed, 686 insertions(+) create mode 100644 tutorials/notebook/synchronization_training_and_evaluation.ipynb create mode 100644 tutorials/source_zh_cn/advanced_use/images/synchronization_training_and_evaluation.png create mode 100644 tutorials/source_zh_cn/advanced_use/synchronization_training_and_evaluation.md diff --git a/tutorials/notebook/README.md b/tutorials/notebook/README.md index fb999501..f7eac5dd 100644 --- a/tutorials/notebook/README.md +++ b/tutorials/notebook/README.md @@ -57,6 +57,7 @@ | 数据处理与数据增强 | [data_loading_enhancement.ipynb](https://gitee.com/mindspore/docs/blob/master/tutorials/notebook/data_loading_enhance/data_loading_enhancement.ipynb) | 使用指南 | - 学习MindSpore中数据处理和增强的方法
- 展示数据处理、增强方法的实际操作
- 对比展示数据处理前和处理后的效果
- 表述在数据处理、增强后的意义 | 自然语言处理应用 | [nlp_application.ipynb](https://gitee.com/mindspore/docs/blob/master/tutorials/notebook/nlp_application.ipynb) | 应用实践 | - 展示MindSpore在自然语言处理的应用
- 展示自然语言处理中数据集特定的预处理方法
- 展示如何定义基于LSTM的SentimentNet网络 | 计算机视觉应用 | [computer_vision_application.ipynb](https://gitee.com/mindspore/docs/blob/master/tutorials/notebook/computer_vision_application.ipynb) | 应用实践 | - 学习MindSpore卷积神经网络在计算机视觉应用的过程
- 学习下载CIFAR-10数据集，搭建运行环境
- 学习使用ResNet-50构建卷积神经网络
- 学习使用Momentum和SoftmaxCrossEntropyWithLogits构建优化器和损失函数
- 学习调试参数训练模型，判断模型精度 +| 模型的训练及验证同步方法 | [synchronization_training_and_evaluation.ipynb](https://gitee.com/mindspore/docs/blob/master/tutorials/notebook/synchronization_training_and_evaluation.ipynb) | 应用实践 | - 了解模型训练和验证同步进行的方法
- 学习同步训练和验证中参数设置方法
- 利用绘图函数从保存的模型中挑选出最优模型 | 使用PyNative进行神经网络的训练调试体验 | [debugging_in_pynative_mode.ipynb](https://gitee.com/mindspore/docs/blob/master/tutorials/notebook/debugging_in_pynative_mode.ipynb) | 模型调优 | - GPU平台下从数据集获取单个数据进行单个step训练的数据变化全过程解读
- 了解PyNative模式下的调试方法
- 图片数据在训练过程中的变化情况的图形展示
- 了解构建权重梯度计算函数的方法
- 展示1个step过程中权重的变化及数据展示 | 自定义调试信息体验文档 | [customized_debugging_information.ipynb](https://gitee.com/mindspore/docs/blob/master/tutorials/notebook/customized_debugging_information.ipynb) | 模型调优 | - 了解MindSpore的自定义调试算子
- 学习使用自定义调试算子Callback设置定时训练
- 学习设置metrics算子输出相对应的模型精度信息
- 学习设置日志环境变量来控制glog输出日志 | MindInsight的模型溯源和数据溯源体验 | [mindinsight_model_lineage_and_data_lineage.ipynb](https://gitee.com/mindspore/docs/blob/master/tutorials/notebook/mindinsight/mindinsight_model_lineage_and_data_lineage.ipynb) | 模型调优 | - 了解MindSpore中训练数据的采集及展示
- 学习使用SummaryRecord记录数据
- 学习使用回调函数SummaryCollector进行数据采集
- 使用MindInsight进行数据可视化
- 了解数据溯源和模型溯源的使用方法 diff --git a/tutorials/notebook/synchronization_training_and_evaluation.ipynb b/tutorials/notebook/synchronization_training_and_evaluation.ipynb new file mode 100644 index 00000000..486fd4cf --- /dev/null +++ b/tutorials/notebook/synchronization_training_and_evaluation.ipynb @@ -0,0 +1,510 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#
同步训练和验证模型体验" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 概述" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "在面对复杂网络时，往往需要进行几十甚至几百次的epoch训练。而在训练之前，往往很难掌握在训练到第几个epoch时，模型的精度能达到满足要求的程度。所以经常会采用一边训练的同时，在相隔固定epoch的位置对模型进行精度验证，并保存相应的模型，等训练完毕后，通过查看对应模型精度的变化就能迅速地挑选出相对最优的模型，本文将采用这种方法，以LeNet网络为样本，进行示例。" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "整体流程如下：\n", + "1. 数据集准备。\n", + "2. 构建神经网络。\n", + "3. 定义回调函数EvalCallBack。\n", + "4. 定义训练网络并执行。\n", + "5. 定义绘图函数并对不同epoch下的模型精度绘制出折线图。" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 数据准备" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 数据集的下载" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "训练数据集下载地址：{\"http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz \", \"http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz \"}。\n", + "\n", + "测试数据集：{\"\", \"\"}\n", + "
数据集放在----*Jupyter工作目录+\\MNIST_Data\\*，如下图结构：" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "```\n", + "MNIST\n", + "├── test\n", + "│   ├── t10k-images-idx3-ubyte\n", + "│   └── t10k-labels-idx1-ubyte\n", + "└── train\n", + " ├── train-images-idx3-ubyte\n", + " └── train-labels-idx1-ubyte \n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 数据集的增强操作" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "下载下来后的数据集，需要通过`mindspore.dataset`处理成适用于MindSpore框架的数据，再使用一系列框架中提供的工具进行数据增强操作来适应LeNet网络的数据处理需求。" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "import mindspore.dataset as ds\n", + "import mindspore.dataset.transforms.vision.c_transforms as CV\n", + "import mindspore.dataset.transforms.c_transforms as C\n", + "from mindspore.dataset.transforms.vision import Inter\n", + "from mindspore.common import dtype as mstype\n", + "\n", + "def create_dataset(data_path, batch_size=32, repeat_size=1,\n", + " num_parallel_workers=1):\n", + " # define dataset\n", + " mnist_ds = ds.MnistDataset(data_path)\n", + "\n", + " # define map operations\n", + " resize_op = CV.Resize((32, 32), interpolation=Inter.LINEAR) \n", + " rescale_nml_op = CV.Rescale(1 / 0.3081, -1 * 0.1307 / 0.3081) \n", + " rescale_op = CV.Rescale(1/255.0, 0.0) \n", + " hwc2chw_op = CV.HWC2CHW() \n", + " type_cast_op = C.TypeCast(mstype.int32) \n", + "\n", + " # apply map operations on images\n", + " mnist_ds = mnist_ds.map(input_columns=\"label\", operations=type_cast_op, num_parallel_workers=num_parallel_workers)\n", + " mnist_ds = mnist_ds.map(input_columns=\"image\", operations=[resize_op,rescale_op,rescale_nml_op,hwc2chw_op],\n", + " num_parallel_workers=num_parallel_workers)\n", + "\n", + " # apply DatasetOps\n", + " buffer_size = 10000\n", + " mnist_ds = mnist_ds.shuffle(buffer_size=buffer_size)\n", + " mnist_ds = mnist_ds.batch(batch_size, drop_remainder=True)\n", + " mnist_ds = mnist_ds.repeat(repeat_size)\n", + " \n", + " return mnist_ds" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 构建神经网络" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "LeNet网络属于7层神经网络，其中涉及卷积层，全连接层，函数激活等算法，在MindSpore中都已经建成相关算子只需导入使用，如下先将卷积函数，全连接函数，权重等进行初始化，然后在LeNet5中定义神经网络并使用`construct`构建网络。" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [], + "source": [ + "import mindspore.nn as nn\n", + "from mindspore.common.initializer import TruncatedNormal\n", + "\n", + "\n", + "def conv(in_channels, out_channels, kernel_size, stride=1, padding=0):\n", + " \"\"\"Conv layer weight initial.\"\"\"\n", + " weight = weight_variable()\n", + " return nn.Conv2d(in_channels, out_channels,\n", + " kernel_size=kernel_size, stride=stride, padding=padding,\n", + " weight_init=weight, has_bias=False, pad_mode=\"valid\")\n", + "\n", + "def fc_with_initialize(input_channels, out_channels):\n", + " \"\"\"Fc layer weight initial.\"\"\"\n", + " weight = weight_variable()\n", + " bias = weight_variable()\n", + " return nn.Dense(input_channels, out_channels, weight, bias)\n", + "\n", + "def weight_variable():\n", + " \"\"\"Weight initial.\"\"\"\n", + " return TruncatedNormal(0.02)\n", + "\n", + "class LeNet5(nn.Cell):\n", + " \"\"\"Lenet network structure.\"\"\"\n", + " # define the operator required\n", + " def __init__(self):\n", + " super(LeNet5, self).__init__()\n", + " self.conv1 = conv(1, 6, 5)\n", + " self.conv2 = conv(6, 16, 5)\n", + " self.fc1 = fc_with_initialize(16 * 5 * 5, 120)\n", + " self.fc2 = fc_with_initialize(120, 84)\n", + " self.fc3 = fc_with_initialize(84, 10)\n", + " self.relu = nn.ReLU()\n", + " self.max_pool2d = nn.MaxPool2d(kernel_size=2, stride=2)\n", + " self.flatten = nn.Flatten()\n", + "\n", + " # use the preceding operators to construct networks\n", + " def construct(self, x):\n", + " x = self.conv1(x)\n", + " x = self.relu(x)\n", + " x = self.max_pool2d(x)\n", + " x = self.conv2(x)\n", + " x = self.relu(x)\n", + " x = self.max_pool2d(x)\n", + " x = self.flatten(x)\n", + " x = self.fc1(x)\n", + " x = self.relu(x)\n", + " x = self.fc2(x)\n", + " x = self.relu(x)\n", + " x = self.fc3(x)\n", + " return x" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 定义回调函数EvalCallBack" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "实现思想：每隔n个epoch验证一次模型精度，由于在自定义函数中实现，如需了解自定义回调函数的详细用法，请参考[API说明](https://www.mindspore.cn/api/zh-CN/master/api/python/mindspore/mindspore.train.html?highlight=callback#mindspore.train.callback.Callback)。\n", + "\n", + "核心实现：回调函数的`epoch_end`内设置验证点，如下：\n", + "\n", + "`cur_epoch % eval_per_epoch == 0`：即每`eval_per_epoch`个epoch结束时，验证一次模型精度。\n", + "\n", + "- `cur_epoch`：当前训练过程的epoch数值。\n", + "- `eval_per_epoch`：用户自定义数值，即验证频次。\n", + "\n", + "其他参数解释：\n", + "\n", + "- `model`：即是MindSpore中的`Model`函数。\n", + "- `eval_dataset`：验证数据集。\n", + "- `epoch_per_eval`：记录验证模型的精度和相应的epoch数，其数据形式为`{\"epoch\":[],\"acc\":[]}`。" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [], + "source": [ + "import matplotlib.pyplot as plt\n", + "from mindspore.train.callback import Callback\n", + "\n", + "class EvalCallBack(Callback):\n", + " def __init__(self, model, eval_dataset, eval_per_epoch):\n", + " self.model = model\n", + " self.eval_dataset = eval_dataset\n", + " self.eval_per_epoch = eval_per_epoch\n", + " \n", + " def epoch_end(self, run_context):\n", + " cb_param = run_context.original_args()\n", + " cur_epoch = cb_param.cur_epoch_num\n", + " if cur_epoch % self.eval_per_epoch == 0:\n", + " acc = self.model.eval(self.eval_dataset,dataset_sink_mode = True)\n", + " epoch_per_eval[\"epoch\"].append(cur_epoch)\n", + " epoch_per_eval[\"acc\"].append(acc[\"Accuracy\"])\n", + " print(acc)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 定义训练网络并执行" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "在保存模型的参数`CheckpointConfig`中，需计算好单个epoch中的step数，再根据需要进行验证模型精度的频次对应，\n", + "本次示例为1875个step/epoch，按照每两个epoch验证一次的思想，这里设置`save_checkpoint_steps=eval_per_epoch*1875`，\n", + "其中变量`eval_per_epoch`等于2。\n", + "\n", + "参数解释：\n", + "\n", + "- `train_data_path`：训练数据集地址。\n", + "- `eval_data_path`：验证数据集地址。\n", + "- `train_data`：训练数据集。\n", + "- `eval_data`：验证数据集。\n", + "- `net_loss`：定义损失函数。\n", + "- `net-opt`：定义优化器函数。\n", + "- `config_ck`：定义保存模型信息。\n", + " - `save_checkpoint_steps`：每多少个step保存一次模型。\n", + " - `keep_checkpoint_max`：设置保存模型数量的上限。\n", + "- `ckpoint_cb`：定义模型保存的名称及路径信息。\n", + "- `model`：定义模型。\n", + "- `model.train`：模型训练函数。\n", + "- `epoch_per_eval`：定义收集`epoch`数和对应模型精度信息的字典。" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": { + "scrolled": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "epoch: 1 step: 375, loss is 2.3058078\n", + "epoch: 1 step: 750, loss is 2.3073978\n", + "epoch: 1 step: 1125, loss is 2.3103657\n", + "epoch: 1 step: 1500, loss is 0.65018296\n", + "epoch: 1 step: 1875, loss is 0.07800862\n", + "epoch: 2 step: 375, loss is 0.010344766\n", + "epoch: 2 step: 750, loss is 0.052723818\n", + "epoch: 2 step: 1125, loss is 0.08183526\n", + "epoch: 2 step: 1500, loss is 0.007430988\n", + "epoch: 2 step: 1875, loss is 0.0076965275\n", + "{'Accuracy': 0.9753605769230769}\n", + "epoch: 3 step: 375, loss is 0.11964749\n", + "epoch: 3 step: 750, loss is 0.04522314\n", + "epoch: 3 step: 1125, loss is 0.018271001\n", + "epoch: 3 step: 1500, loss is 0.006928641\n", + "epoch: 3 step: 1875, loss is 0.15374172\n", + "epoch: 4 step: 375, loss is 0.12120275\n", + "epoch: 4 step: 750, loss is 0.122824214\n", + "epoch: 4 step: 1125, loss is 0.0023852547\n", + "epoch: 4 step: 1500, loss is 0.018273383\n", + "epoch: 4 step: 1875, loss is 0.08102103\n", + "{'Accuracy': 0.9821714743589743}\n", + "epoch: 5 step: 375, loss is 0.12944886\n", + "epoch: 5 step: 750, loss is 0.0010141768\n", + "epoch: 5 step: 1125, loss is 0.0054096584\n", + "epoch: 5 step: 1500, loss is 0.0022614016\n", + "epoch: 5 step: 1875, loss is 0.07229582\n", + "epoch: 6 step: 375, loss is 0.0025749032\n", + "epoch: 6 step: 750, loss is 0.06261393\n", + "epoch: 6 step: 1125, loss is 0.021273317\n", + "epoch: 6 step: 1500, loss is 0.011360342\n", + "epoch: 6 step: 1875, loss is 0.12855275\n", + "{'Accuracy': 0.9853766025641025}\n", + "epoch: 7 step: 375, loss is 0.09330422\n", + "epoch: 7 step: 750, loss is 0.002063415\n", + "epoch: 7 step: 1125, loss is 0.0047940286\n", + "epoch: 7 step: 1500, loss is 0.0052507296\n", + "epoch: 7 step: 1875, loss is 0.018066114\n", + "epoch: 8 step: 375, loss is 0.08678668\n", + "epoch: 8 step: 750, loss is 0.02440551\n", + "epoch: 8 step: 1125, loss is 0.0017507032\n", + "epoch: 8 step: 1500, loss is 0.02957578\n", + "epoch: 8 step: 1875, loss is 0.0023948685\n", + "{'Accuracy': 0.9863782051282052}\n", + "epoch: 9 step: 375, loss is 0.012376097\n", + "epoch: 9 step: 750, loss is 0.029711302\n", + "epoch: 9 step: 1125, loss is 0.017438065\n", + "epoch: 9 step: 1500, loss is 0.015443239\n", + "epoch: 9 step: 1875, loss is 0.0031764025\n", + "epoch: 10 step: 375, loss is 0.0005294987\n", + "epoch: 10 step: 750, loss is 0.0015696918\n", + "epoch: 10 step: 1125, loss is 0.019949459\n", + "epoch: 10 step: 1500, loss is 0.004248183\n", + "epoch: 10 step: 1875, loss is 0.07389321\n", + "{'Accuracy': 0.9824719551282052}\n" + ] + } + ], + "source": [ + "from mindspore.train.serialization import load_checkpoint, load_param_into_net\n", + "from mindspore.train.callback import ModelCheckpoint, CheckpointConfig, LossMonitor\n", + "from mindspore.train import Model\n", + "from mindspore import context\n", + "from mindspore.nn.metrics import Accuracy\n", + "from mindspore.nn.loss import SoftmaxCrossEntropyWithLogits\n", + "\n", + "if __name__ == \"__main__\":\n", + " context.set_context(mode=context.GRAPH_MODE, device_target=\"GPU\")\n", + " train_data_path = \"./MNIST_Data/train\"\n", + " eval_data_path = \"./MNIST_Data/test\"\n", + " ckpt_save_dir = \"./lenet_ckpt\"\n", + " epoch_size = 10\n", + " eval_per_epoch = 2\n", + " repeat_size = 1\n", + " network = LeNet5()\n", + " \n", + " train_data = create_dataset(train_data_path,repeat_size = repeat_size)\n", + " eval_data = create_dataset(eval_data_path,repeat_size = repeat_size)\n", + " \n", + " # define the loss function\n", + " net_loss = SoftmaxCrossEntropyWithLogits(is_grad=False, sparse=True, reduction='mean')\n", + " # define the optimizer\n", + " net_opt = nn.Momentum(network.trainable_params(), learning_rate=0.01, momentum=0.9)\n", + " config_ck = CheckpointConfig(save_checkpoint_steps=eval_per_epoch*1875, keep_checkpoint_max=15)\n", + " ckpoint_cb = ModelCheckpoint(prefix=\"checkpoint_lenet\",directory=ckpt_save_dir, config=config_ck)\n", + " model = Model(network, net_loss, net_opt, metrics={\"Accuracy\": Accuracy()})\n", + " \n", + " epoch_per_eval = {\"epoch\":[],\"acc\":[]}\n", + " eval_cb = EvalCallBack(model,eval_data,eval_per_epoch)\n", + " \n", + " model.train(epoch_size, train_data, callbacks=[ckpoint_cb, LossMonitor(375),eval_cb],\n", + " dataset_sink_mode=True)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "在同一目录的文件夹中可以看到`lenet_ckpt`文件夹中，保存了5个模型，和一个计算图相关数据，其结构如下：" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "```\n", + "lenet_ckpt\n", + "├── checkpoint_lenet-10_1875.ckpt\n", + "├── checkpoint_lenet-2_1875.ckpt\n", + "├── checkpoint_lenet-4_1875.ckpt\n", + "├── checkpoint_lenet-6_1875.ckpt\n", + "├── checkpoint_lenet-8_1875.ckpt\n", + "└── checkpoint_lenet-graph.meta\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 绘制不同epoch下模型的精度" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "定义绘图函数`eval_show`，将`epoch_per_eval`载入到`eval_show`中，绘制出不同`epoch`下模型的验证精度折线图。" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEWCAYAAABxMXBSAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3debzWY/7H8ddb0SLZygyisgxiZEiWkMk+wxhb9m1sQxnrjzBmyJKxm9Ega3ZjnRgVmsrUYDohIhQmEsqWpRHV5/fH9T26O85yn5z7fM/yfj4e53Hu+/4u9+c+y/25r+v6Xp9LEYGZmVmxlso7ADMza1ycOMzMrFacOMzMrFacOMzMrFacOMzMrFacOMzMrFacOKzkJHWRFJJaFrHvEZLG1UdczYWkgyU98QOOHy7p8LqMqYbnK/rvxfLhxGGLkfRfSd9I6lDh8Rezf+Yu+URmSyoi7oqInYvZV9J5ku6scPxuETG0NNHVj+xvd52842gqnDisMm8DB5bfkfRToE1+4TQMjfETcGOMuS4199dfKk4cVpk7gMMK7h8O3F64g6TlJd0uabak6ZJ+L2mpbFsLSZdL+kjSW8AvKzn2ZknvS3pP0oWSWhQTmKT7JX0gaY6kpyVtWLCtjaQrsnjmSBonqU22bRtJ/5b0maR3JR2RPT5G0tEF51isqyz7pNpP0lRgavbYNdk5Ppc0UdK2Bfu3kHS2pDclfZFtX0PSYElXVHgtj0o6uZLXeL2kyys89ndJp2a3BxSc/1VJe1WIf7ykqyR9ApxXyWuqNH5JuwJnA/tL+lLSpIo/I0lLZb/r6ZJmZX8Dy2fbyruYDpf0Tvb7P6ea32WVv6/MwZWdR1JPSc9kv8v3JV0raZmqfmeSns42Tcpe1/5VxWRFigh/+eu7L+C/wI7A68AGQAvgXaAzEECXbL/bgb8DywFdgDeAo7JtvwVeA9YAVgJGZ8e2zLY/AtwALAusAvwHOC7bdgQwrpr4fpM9ZyvgauDFgm2DgTHA6lncW2f7rQl8QWpFLQ2sDGySHTMGOLrgHIs9fxb3k9nraJM9dkh2jpbAacAHQOts2/8BLwPrAQK6Z/v2BGYCS2X7dQDmAj+q5DVul/3Mld1fEfgfsFp2fz9gNdIHv/2Br4BVC+KfD5yYxdemktdUXfznAXdWiOe7n1H2858GrAW0Ax4C7si2dcl+Xjdmz9sdmAdsUMXvsqrfV7XnATYDtszi7wJMAU6u4XcWwDp5/381la/cA/BXw/piUeL4PTAI2DX7J2yZ/fN1yf7J5wHdCo47DhiT3f4n8NuCbTtnx7YEfpQd26Zg+4HA6Oz2Ym9yNcS6Qnbe5bM30f8B3SvZ7yzg4SrO8d2bYmXPn52/Tw1xfFr+vKSEu2cV+00Bdspu9wcer2I/Ae8A22X3jwH+Wc3zv1j+nFn871TYXu3PtEL851F94hgFnFCwbT3g24I38QA6FWz/D3BAJc9Z3e+r6PNk204u/P1W9jvDiaNOv9xVZVW5AziI9KZze4VtHYBlgOkFj00nfXKE9Gn43QrbynUmfep/P+tq+IzU+lilpoCybqBLsm6az0lJrjyeDkBr4M1KDl2jiseLVfhakHSapClZ98pnpMRVfjFBdc81lPRpn+z7HZXtFOmd7l4WjTMdBNxV8PyHKV2sUP7z26jg+b8Xb0U1xF+T1fj+7738A0G5DwpuzyW1TCqq7vdV7Xkk/UTSY1mX5efAxZXEX+3PwH4YJw6rVERMJw2S/4LUHVHoI9KnzM4Fj60JvJfdfp/0Blq4rdy7pBZHh4hYIftqHxEbUrODgD1JLaLlSZ9MIX1C/wj4Gli7kuPereJxSN08bQvu/7iSfb4rIZ2NB5wJ9AVWjIgVgDlZDDU9153AnpK6k7oBH6liP4B7gH0ldQa2AB7Mnr8zqQunP7By9vyTC55/sXgrKiL+msplz+T7v/f5wIc1HFdRdb+vmlxH6gpdNyLak8ZlVGEfl/0uIScOq85RpCb/V4UPRsQC4G/ARZKWy97MTiW9MZJt+52kTpJWBAYUHPs+8ARwhaT22WDr2pJ6FxHPcqSk8zHpzf7igvMuBG4BrpS0WtY62UpSK9Kn9R0l9ZXUUtLKkjbJDn0R2FtSW6XLNY8qIob5wGygpaQ/AO0Ltt8EXCBpXSUbS1o5i3EGMIHU0ngwIv5X1ZNExAvZc9wEjIyIz7JNy5LeFGcDSDqS1OIoVk3xfwh0UXahQyXuAU6R1FVSO9Lv4L6ImF+LGGr6fRXzGj4HvpS0PnB8Ecd8SBqXsTrgxGFViog3I6Ksis0nkj6tvwWMA+4mvRFA+kQ8EpgEPM/3WyyHkbq6XiX1rz8ArFpESLeTukbey459tsL200kD0xOAT4A/kQaj3yG1nE7LHn+RNOAKcBXwDemNZSgFXUJVGAkMJ10MMJ30qbmwW+RKUuJ8gvTmdjOLX8o8FPgpVXRTVXAPqXV1d/kDEfEqcAXwTBbzT4HxRZyr2Pjvz75/LOn5So6/JYv9aVKL9GvS38KSqPT3VeRxB5EueLgRuK+IY84Dhmbde32XKFr7TvlVG2ZWDyRtR2qZdck+dZs1Om5xmNUTSUsDJwE3OWlYY+bEYVYPJG0AfEbqkrs653DMfhB3VZmZWa24xWFmZrXSLAqAdejQIbp06ZJ3GGZmjcrEiRM/ioiOFR9vFomjS5culJVVdVWpmZlVRtL0yh53V5WZmdWKE4eZmdWKE4eZmdWKE4eZmdWKE4eZmdWKE4eZmdWKE4eZmdVKs5jHYWaN3MyZMG4cfPUVHHYYtGiRd0TNmhOHmTUsEfD66ylR/Otf6ftbby3a/thjcNdd0Lp1fjE2c04cZpavb7+FF15YPFF89FHa1rEjbLMN9O+fvo8bB6eeCr/8JTzyCCy3XL6xN1NOHGZWv778Ep59dlGiePZZmDs3bVt77ZQUtt02JYqf/ARUsJz45ptDhw5w5JHQpw88/nhKLlavnDjMrLQ+/BDGj1/UmnjhBViwAJZaCrp3h6OPTkmiVy9YbbWaz3foobDiirDffinBPPEErLlm6V+HfaekiUPSrsA1QAvSqmeXVNjembSGcUfSmsOHRMSMbNulwC9JV349CZwUESFpGeBaYHtgIXBORDxYytdhZkWKgDffXLzb6Y030rbWrWGLLeCss1Ki2GoraN9+yZ5n991Twthjj5RwnngCNtig7l6HVatkiUNSC2AwsBMwA5ggaVhEvFqw2+XA7RExVFIfYBBwqKStgV7Axtl+44DewBjgHGBWRPxE0lLASqV6DWZWg/nz4aWXFiWJcePggw/StpVWSm/q5S2KzTaDZZapu+fedlsYOxZ22SXdfvxx6Nmz7s5vVSpli6MnMC0i3gKQdC+wJ1CYOLoBp2S3RwOPZLcDaA0sAwhYGvgw2/YbYH2AbN3mj0r3EsxsMXPnwn/+syhRPPMMfPFF2ta5M+y4Y0oS224L66+fuqNKqXv31A22005pzOORR1IMVlKlTByrA+8W3J8BbFFhn0nAPqTurL2A5SStHBHPSBoNvE9KHNdGxBRJK2THXSBpe+BNoH9EfFjhvEg6FjgWYE33f5otmY8/Xnx8YuLEdBWUBD/9aRpv2Gab9LXGGvnEuPbaKcZddoFf/ALuvhv23TefWJqJUiYOVfJYxQXOTweulXQE8DTwHjBf0jrABkCnbL8nJW1Haq10AsZHxKmSTiV1dx36vSeKGAIMAejRo4cXVjerSQRMn74oSfzrXzBlStq2zDKpG+i001JrYqut0gB1Q7Hqqqnbao89oG9fuO46OO64vKNqskqZOGYAhR9BOgEzC3eIiJnA3gCS2gH7RMScrLXwbER8mW0bDmwJ/AuYCzycneJ+4KgSvgazpmvBApg8edHYxL/+Be+9l7Ytv3wanzj00JQoevRo+BPuVlwxDZLvtx/89reptXTWWYtfzmt1opSJYwKwrqSupJbEAcBBhTtI6gB8ko1VnEW6wgrgHeAYSYNILZfewNXZVVWPkq6o+iewA4uPmZhZVb7+GiZMWJQk/v1vmDMnbVt99ZQgyudPbLhh4yzr0bZtGuc48kg455w0kfDyy0s/1tLMlCxxRMR8Sf2BkaTLcW+JiFckDQTKImIYKQEMkhSkrqp+2eEPAH2Al0ndWyMi4tFs25nAHZKuBmYDR5bqNZg1ap99lvr+yxPFhAnwzTdpW7dusP/+ixJF585N55P50kvD7benq7quuiq1PG66KT1udUIRTb/7v0ePHlFWVpZ3GGal9e67i8+fmDw5jVu0bJm6msqvdtp66zT7uqmLgAsvhD/8IY193HcftGmTd1SNiqSJEdGj4uOeOW7WGC1cmAauCxPF9OlpW7t2KTn07ZuSRc+eqQunuZHg3HNTkuzXL1119eijafzGfhAnDrPG4Jtv0qWw5Yli/Hj45JO07cc/Tgni1FPT9403Tq0MS44/PnVbHXoo9O4NI0akn5ktMf91mTVEn3+eJteVtyaeey4NbkMq/LfXXou6ntZaq+mMT5TK/vvDCivA3nunn9uTT0LXrnlH1Wg5cZg1BO+/v3i306RJqTuqRQv42c/Sp+byiXarrJJ3tI3TLrvAqFFpkmCvXjByZJrEaLXmxGGWl4kT4S9/SYnizTfTY23bpsl1556bksSWW6YxC6sbW26ZkvPOO8N228E//pHGg6xWnDjM6lsEXHttmoXdrh1svz2ccELqdtpkE182WmobbpjGiHbeOdW1euCB1AqxojlxmNWnOXNStdgHHkiXiN52Wxq4tfrVpUtq6e26K+y5JwwdCgcdVONhlng6pVl9eeGFVFr84Yfhssvg73930sjTKqvAmDFpvOPgg1O3oRXFicOs1CLghhvS2MXXX6difKef7iuhGoL27dPlub/+Nfzud/DHP6bfl1XLicOslL78Eg45JBXd23771Oro1SvvqKxQ69Zw//2pvtXAgdC/f7qizarkMQ6zUnn55VSpdepUuOgiGDDAxfYaqpYt4eab0yzzyy5LkyuHDq3bFQubECcOs1K49dZU5mL55dPcge23zzsiq4kEl16akseZZ8Knn8KDD8Kyy+YdWYPjjz9mdWnu3NTl8ZvfpPkBL77opNHYnHFGqqb75JNpSdry0i72HScOs7oyZUoqKDh0aBpkHTkSfvSjvKOyJXHUUemS6YkT00TB8gWuDHDiMKsbd94Jm28Os2alhHHeeY1zISRbZK+90hVX06enWfxTp+YdUYPhxGH2Q/zvf3Dssany6mabpa6pnXbKOyqrKz//OYwena6O22abdFWcOXGYLbGpU9PcjBtvTGtbjxoFq62Wd1RW13r0SLPMW7VK41Vjx+YdUe6cOMyWxN/+lloY776bCuVdfLHXwGjK1lsv1bdaffVUZXfYsLwjypUTh1ltzJuXJojtvz9stFHqmnKBvOZhjTVSZd3u3dO6HrfdlndEuXHiMCvW22+nWd+DB6fKtmPHpjcTaz5WXjl1Sfbpky67vuKKvCPKhROHWTEeeSQtqPTmm+n25Ze7/Hlz1a5dWrt8v/1SzbGzzmp29a3cKWtWnW++SaVCrroqXW57331ectTSQPk996TqxpdcAh99BNdf32wuwXbiMKvKO+9A375pve8TT0w1jFq1yjsqayhatIDrroOOHeHCC9MM87vuSkUTmzgnDrPK/OMfcNhh8O23qXLqvvvmHZE1RBJccEEa+zjlFPjlL1NX5nLL5R1ZSXmMw6zQ/Pmpa2r33WHNNeH55500rGYnnwy3354umOjTB2bPzjuiknLiMCv33ntppvCf/gTHHQfPPAPrrJN3VNZYHHpoam1MnpzWj3/nnbwjKhknDjOAJ56ATTZJJSXuvjsNdDaDvmqrY7vvnv6WPvggXbo9ZUreEZVESROHpF0lvS5pmqQBlWzvLGmUpJckjZHUqWDbpZJekTRF0p+lxdfZlDRM0uRSxm/NwIIF8Ic/wK67wo9/DGVlcOCBeUdljdm226Yuq2+/TbcnTMg7ojpXssQhqQUwGNgN6AYcKKlbhd0uB26PiI2BgcCg7NitgV7AxsBGwOZA74Jz7w18WarYrZn44INUkPCCC+CII9LVU+uvn3dU1hR0755KlLRvn7o/n3oq74jqVClbHD2BaRHxVkR8A9wL7Flhn27AqOz26ILtAbQGlgFaAUsDHwJIagecClxYwtitqRs9OnVNPftsKh1xyy3Qtm3eUVlTsvbaKXmstVa62uqBB/KOqM6UMnGsDrxbcH9G9lihScA+2e29gOUkrRwRz5ASyfvZ18iIKO8svAC4Aphb3ZNLOlZSmaSy2U38CgerhYUL0zX3O+4IK64I//kPHH543lFZU7XqqqnbavPN05ygIUPyjqhOlDJxqJLHKs7LPx3oLekFUlfUe8B8SesAGwCdSMmmj6TtJG0CrBMRD9f05BExJCJ6RESPjh07/qAXYk3E7Nmw225w7rlpHGPChFSo0KyUVlwxDZjvtlu6Wm/QoEZfoqSUEwBnAIUV4DoBMwt3iIiZwN7wXRfUPhExR9KxwLMR8WW2bTiwJfAFsJmk/2axryJpTERsX8LXYU3BuHFwwAGpNMSQIXD00Wnylll9aNs2Xap75JFw9tnpQ8zll8NSjfPC1lJGPQFYV1JXScsABwCLFbGX1EFSeQxnAbdkt98htURaSlqa1BqZEhHXRcRqEdEF2AZ4w0nDqrVwYZqXsf320KZNGtM45hgnDat/Sy+dJgmeeGKqfXbkkenKq0aoZC2OiJgvqT8wEmgB3BIRr0gaCJRFxDBge2CQpACeBvplhz8A9AFeJnVvjYiIR0sVqzVRH3+cxi/+8Y/Uv3zjjekqF7O8LLUUXHNNqm/1hz/Ap5+mwplt2uQdWa0oGnlfWzF69OgRZWVleYdh9enZZ9NiSx98kD7dHX+8WxnWsFx3HfTrl9Yyf/RRWH75vCP6HkkTI6JHxccbZwebWVUiUqLYdttUvXT8eDjhBCcNa3iOPz6VZn/2WejdO33IaSScOKzp+Owz2GcfOPXUVPrh+eehx/c+LJk1HPvvn1obU6emlsfbb+cdUVGcOKxpmDgRNt00/RNedRU89BCssELeUZnVbJdd0nK0n3yS6lu9/HLeEdXIicMat4i0BvjWW6eS6P/6Vypx7a4pa0y23DL97Uqw3Xbw73/nHVG1nDis8fr88zQ3o3//VHPqhRfSP6BZY7ThhmlMrmPHVNlg+PC8I6qSE4c1TpMmpfGLBx9M8zSGDUursJk1Zl26pMmq668Pv/pVGjxvgJw4rHGJSPMxttgCvvoKxoyBM85otDNwzb5nlVXS33WvXnDwwXDttXlH9D3+b7PG48sv0zrgxx6bLl984YV0JYpZU9O+PYwYAXvumWaan3deg6pv5cRhjcMrr6QKo3ffndbPGD48fTIza6pat4b770+lSc4/PyWQhQvzjgoobZFDs7oxdGiaLNW+fVoQ5+c/zzsis/rRsiXcfDN06ACXXZYu2b3tNlhmmXzDyvXZzaozd276lHXLLalI4T33pOVdzZoTCS69NCWPM89M9a0eeACWXTa3kNxVZQ3T66+nAfBbb03rZzz1lJOGNW9nnAE33ZTW9thpp9T6yIkThzU8d98Nm22WaveMGAEDB6a6U2bN3VFHpdbGxIlpouB77+USRo2JI1t+tZ+kFesjIGvGvv4afvvbdAniz34GL74IO++cd1RmDctee6UPVNOnp6sKp06t9xCKaXEcAKwGTJB0r6RdJNdzsDo2bRpstRXccEPqxx09GlavuES9mQHpApHRo9Ml6ttsky5Nr0c1Jo6ImBYR5wA/Ae4mrdL3jqTzJa1U6gCtGXjwwdQ1NX16KlJ4ySXpahIzq1qPHmmWeatW6eKRsWPr7amLGuOQtDFwBXAZ8CCwL/A58M/ShWZN3rx58Lvfwb77QrduqWtq993zjsqs8VhvvVTfavXVU5XdYcNqPqYOFDPGMRG4irSG+MYR8buIeC4irgDeKnWA1kS9/XZabOkvf4FTTkmfltZcM++ozBqfNdZIlXW7d4e9907znkqsmP6A/SKi0gQREXvXcTzWHAwbltYCj0jrZuy1V94RmTVuK6+c1vTYe2844gj4+OO0oFmJFNNVdbSk71bEkbSipAtLFpE1Xd9+C6efnurvrL12WqHPScOsbrRrl8YI99sPTjsNzj67ZPWtikkcu0XEZ+V3IuJT4BclicaarnffTYUJr7gC+vVL/bJrrZV3VGZNS6tWqcLCccfBoEHp+4IFdf40xSSOFpJald+R1AZoVc3+ZosbPjzNy5g8Ge67L5WJbuU/IbOSaNECrrsOfv/7VHnh+efr/CmKSRx3AqMkHSXpN8CTQOlHX6zxmz8/NZd/8Yt01UdZGfTtm3dUZk2flKpIT56cqkrXsRoHxyPiUkkvAzsAAi6IiJF1Hok1LTNnwoEHwtNPwzHHwDXXQJs2eUdl1ryst15JTlvULKuIGA403AVwrWF56ik46KC0Qt8dd8Ahh+QdkZnVoWLmcWwpaYKkLyV9I2mBpM/rIzhrZBYsSCuV7bxzWmSprMxJw6wJKmaM41rgQGAq0AY4GvhLKYOyRuqEE9JKZYcdBs89BxtskHdEZlYCRZUciYhpQIuIWBARtwJFLcEmaVdJr0uaJmlAJds7Sxol6SVJYyR1Kth2qaRXJE2R9GclbSX9Q9Jr2bZLin2hVmJjx8KQIWnS0W235brIjJmVVjGJY66kZYAXszfzU4Aa3xUktQAGA7sB3YADJXWrsNvlwO0RsTEwEBiUHbs10AvYGNgI2BzoXX5MRKwP/AzoJWm3Il6DldK8eakceteu6UoOM2vSikkch2b79Qe+AtYA9iniuJ7AtIh4KyK+Ae4F9qywTzdgVHZ7dMH2AFoDy5DmjCwNfBgRcyNiNEB2zueBTli+Lr0UXnsNBg+Gtm3zjsbMSqzaxJG1Gi6KiK8j4vOIOD8iTs26rmqyOvBuwf0Z2WOFJrEoCe0FLCdp5Yh4hpRI3s++RkbElAqxrQDswaLEUzH2Y7NFqMpmz55dRLi2RKZOhYsuSvMzdnPjz6w5qDZxRMQCoGPWVVVblS32VLFwyulAb0kvkLqi3gPmS1oH2IDUmlgd6CNpu+9OLLUE7gH+XE0BxiER0SMienTs2HEJwrcaRaQB8Vat4Oqr847GzOpJMfM4/guMlzSM1FUFQERcWcNxM0jdWuU6ATMLd4iImcDeAJLaAftExBxJxwLPRsSX2bbhwJbA09mhQ4CpEeF3qzzdfXeaszF4MKy6at7RmFk9KWaMYybwWLbvcgVfNZkArCupa9ZiOQBYbJURSR0klcdwFml1QYB3SC2RlpKWJrVGpmTHXAgsD5xcRAxWKp98kq6g6tkzFVIzs2ajmJIj5y/JiSNivqT+wEigBXBLRLwiaSBQFhHDgO2BQZKC1Jrolx3+ANAHeJnUvTUiIh7NLtc9B3gNeD5b+vzaiLhpSWK0H2DAgFTzf+TIVFTNzJoNRQ312iWN5vtjE0REn1IFVdd69OgRZWVleYfRdIwfD9tsk2r+X3553tGYWYlImhgRPSo+XswYx+kFt1uTroKaX1eBWSPz7bdpzsYaa6TyImbW7BTTVTWxwkPjJY0tUTzW0F1xRSrVPGxYWnHMzJqdGhOHpJUK7i4FbAb8uGQRWcP11lswcGBa7nWPPfKOxsxyUkxX1UTSGIdIXVRvA0eVMihrgCLSkq8tWsCf/5x3NGaWo2K6qrrWRyDWwN1/P4wYkSb6dXKVF7PmrJj1OPpl5T3K768o6YTShmUNypw5cNJJsOmm0L9/3tGYWc6KmQB4TER8Vn4nIj4FjildSNbgnH02zJqVyqZ7zoZZs1dM4lhK2Uw7+K7w4ZLUrrLG6Lnn4LrrUktjs83yjsbMGoBiBsdHAn+TdD1pkPy3wIiSRmUNw/z5qZzIqqt6nQ0z+04xieNM4FjgeNKVVU8ALvHRHFxzDUyaBA8+CO3b5x2NmTUQxSSONsCNEXE9fNdV1QqYW8rALGfTp8Mf/gC7757mbZiZZYoZ4xhFSh7l2gBPlSYcaxAi4MQT0+1rrwVVtrSKmTVXxbQ4WpeviwEQEV9K8vqgTdkjj8Cjj8Jll0HnznlHY2YNTDEtjq8kbVp+R9JmwP9KF5Ll6osvUmtj443T3A0zswqKaXGcDNwvqXz1vlWB/UsXkuXq3HNh5kx44AFYeum8ozGzBqiYkiMTJK0PrEe6quq1iPi25JFZ/Zs4Ef7yl1Q2fcst847GzBqoYlockJJGN9J6HD+TRETcXrqwrN4tWJDmbKyyClx8cd7RmFkDVkxZ9T+SlnjtBjwO7AaMA5w4mpLBg1OL4957YYUVat7fzJqtYgbH9wV2AD6IiCOB7qR5HNZUzJgB55wDu+wCffvmHY2ZNXDFJI7/RcRCYL6k9sAsYK3ShmX16qSTUnmRv/7VczbMrEbFjHGUZWXVbyQt6vQl8J+SRmX159FH4aGH0rjGWv48YGY1U0QUv7PUBWgfES+VKqBS6NGjR5SVleUdRsPz1VfQrRsstxw8/zws46LHZraIpIkR0aPi48VeVQVARPy3ziKy/J13HrzzDowb56RhZkUrZozDmqJJk+Cqq+Doo6FXr7yjMbNGxImjOSqfs7HSSvCnP+UdjZk1MlV2VUlaqboDI+KTug/H6sWQIWllvzvuSMnDzKwWqhvjmEha8a+y6zMDX5LbOL3/PgwYADvsAAcfnHc0ZtYIVdlVFRFdI2Kt7HvFr6KShqRdJb0uaZqkAZVs7yxplKSXJI2R1Klg26WSXpE0RdKfy9c9l7SZpJezc373uBXplFNg3ry0jrh/dGa2BGoc41ByiKRzs/trSupZxHEtgMGkEiXdgAMldauw2+XA7RGxMTAQGJQduzXQC9gY2AjYHOidHXMdaSnbdbOvXWuKxTIjRsB998HZZ8O66+YdjZk1UsUMjv8V2Ao4KLv/BSkh1KQnMC0i3oqIb4B7gT0r7NONtMIgwOiC7UEqqLgMqbzJ0sCHklYlzSN5JtIElNuBXxcRi82dCyecAOutB2eemXc0ZtaIFZM4toiIfsDXABHxKekNvSarA+8W3J+RPVZoErBPdnsvYDlJK0fEM6RE8n72NTIipmTHzyDc73oAABM2SURBVKjhnABIOlZSmaSy2bNnFxFuE3fhhfD223D99dDKpcbMbMkVkzi+zbqdAkBSR2BhEcdVNahe6HSgt6QXSF1R75FqYq0DbAB0IiWGPpK2K/Kc6cGIIRHRIyJ6dOzYsYhwm7DJk9MysEccAdtvn3c0ZtbIFZM4/gw8DKwi6SJSSfViFmyYAaxRcL8TMLNwh4iYGRF7R8TPgHOyx+aQWh/PRsSX2Xrnw4Ets3N2qu6cVsHChWlhpuWXT8nDzOwHqjFxRMRdwBmkgev3gV9HxP1FnHsCsK6krpKWAQ4AhhXuIKmDpPIYzgJuyW6/Q2qJtJS0NKk1MiUi3ge+kLRldjXVYcDfi4il+br5Zhg/PiWNDh3yjsbMmoBiJwDOAu4p3FbTBMCImC+pPzASaAHcEhGvSBoIlEXEMNICUYMkBfA00C87/AGgD/AyqStqREQ8mm07HrgNaENqiQwv7qU2Q7NmpYHw7bZL3VRmZnWgyuq4kt5m0QTANYFPs9srAO9ERNf6CvKHarbVcQ89NF1++9JLsP76eUdjZo1MVdVxa5wASGox7BERHSJiZWB34KHShWp14qmn4M470yxxJw0zq0PFDI5vHhGPl9+JiOEsmoxnDdHXX8Pxx8M666TJfmZmdaiY9Tg+kvR74E5S19UhwMcljcp+mIsvhmnT4MknoXXrvKMxsyammBbHgUBH0iW5jwCrZI9ZQ/Taa3DJJamA4Y475h2NmTVBNbY4squnTpLUHliYzauwhigizdlYdlm44oq8ozGzJqqYIoc/zWZ2vwy8ImmipI1KH5rV2tChMHYsXHop/OhHeUdjZk1UMV1VNwCnRkTniOgMnAYMKW1YVmsffQSnn56WgT3qqLyjMbMmrJjEsWxEjC6/ExFjgGVLFpEtmf/7P5gzJxUxXMorAptZ6RRzVdVb2Vocd2T3DwHeLl1IVmtjx8Jtt6U5Gxu5F9HMSquYj6a/IV1V9RDpyqqOwJGlDMpqYd68NCDetSuce27e0ZhZM1DMVVWfAr+rh1hsSVx6aboE9/HHoW3bvKMxs2aguiKHw6raBhARv6r7cKxWpk6Fiy6Cvn1ht93yjsbMmonqWhxbkVbwuwd4jsoXUbK8RKSyIq1awdVX5x2NmTUj1SWOHwM7kWaJHwT8A7gnIl6pj8CsBnffDaNGweDBsOqqeUdjZs1IddVxF0TEiIg4nLT63jRgjKQT6y06q9wnn8Cpp0LPnnDccXlHY2bNTLWD45JaAb8ktTq6kJaRdUn1vA0YAB9/DCNHQosWeUdjZs1MdYPjQ4GNSCvsnR8Rk+stKqva+PFw441w2mmwySZ5R2NmzVB1KwAuBL7K7hbuJCAion2JY6szTWYFwG++gU03hc8/h1dfhXbt8o7IzJqwqlYArLLFERGuW9HQXHklvPIKDBvmpGFmuXFyaCzeegsGDoS99oI99sg7GjNrxpw4GoMI6NcvDYT/+c95R2NmzVwxRQ4tb/ffDyNGpIl+nTrlHY2ZNXNucTR0n30GJ52UBsX79887GjMztzgavHPOgVmz4LHHPGfDzBoEtzgasueeg+uuSy2NzTbLOxozM8CJo+GaPz+VE1ltNbjggryjMTP7TkkTh6RdJb0uaZqkAZVs7yxplKSXJI2R1Cl7/OeSXiz4+lrSr7NtO0h6Pnt8nKR1SvkacnPNNTBpUrqKqn2jmWtpZs1AlTPHf/CJpRbAG6QKuzOACcCBEfFqwT73A49FxFBJfYAjI+LQCudZiVRgsVNEzJX0BrBnREyRdALQMyKOqC6WRjdzfPp06NYN+vRJk/3kivZmVv+qmjleyhZHT2BaRLwVEd8A9wJ7VtinGzAquz26ku0A+wLDI2Judj+A8o/gywMz6zTqvEXAiVkB4muvddIwswanlIljddJCUOVmZI8VmgTsk93eC1hO0soV9jmAtJhUuaOBxyXNAA4FLqmziBuCRx6BRx+F88+Hzp3zjsbM7HtKmTgq+6hcsV/sdKC3pBeA3sB7wPzvTiCtCvwUGFlwzCnALyKiE3ArcGWlTy4dK6lMUtns2bOX/FXUpy++SK2N7t3T3A0zswaolPM4ZgBrFNzvRIVupYiYCewNIKkdsE9EzCnYpS/wcER8m+3TEegeEc9l2+8DRlT25BExBBgCaYzjB7+a+nDuuTBzJjz4ICy9dN7RmJlVqpQtjgnAupK6SlqG1OU0rHAHSR0klcdwFnBLhXMcyOLdVJ8Cy0v6SXZ/J2BKnUeeh4kT4S9/SeuIb7FF3tGYmVWpZC2OiJgvqT+pm6kFcEtEvCJpIFAWEcOA7YFBkgJ4GuhXfrykLqQWy9gK5zwGeDBbL+RT4Deleg31ZsGCNGdjlVXg4ovzjsbMrFolLTkSEY8Dj1d47A8Ftx8AHqji2P/y/cF0IuJh4OE6DTRvgwenFse998Lyy+cdjZlZtTxzPG8zZqR6VLvuCn375h2NmVmNnDjydtJJqbzI4MGes2FmjYKr4+bp0UfhoYdg0CBYa628ozEzK4pbHHn56qtU9XbDDeG00/KOxsysaG5x5OW88+Cdd2DcOM/ZMLNGxS2OPEyaBFddBcccA7165R2NmVmtOHHUt/I5GyutBJc0rTJbZtY8uKuqvt1wQ1rZ7847U/IwM2tk3OKoT++/D2edBTvuCAcdlHc0ZmZLxImjPp1yCsybB3/9q+dsmFmj5cRRX0aMgPvuS7PE110372jMzJaYE0d9mDsXTjgB1lsPzjgj72jMzH4QD47XhwsugLffhjFjoFWrvKMxM/tB3OIotcmT4fLL4YgjoHfvvKMxM/vBnDhKaeFC+O1vU6n0yy7LOxozszrhrqpSuvlmGD8ebr0VOnTIOxozszrhFkepfPhhGgjv3RsOPzzvaMzM6owTR6mcdlqqgHv99Z6zYWZNihNHKTz1FNx1FwwYAOuvn3c0ZmZ1yomjrn39NRx/PKyzDpx9dt7RmJnVOQ+O17WLL4Zp0+DJJ6F167yjMTOrc25x1KXXXkul0g8+OBUyNDNrgpw46kpEmrPRrh1ceWXe0ZiZlYy7qurK0KEwdiwMGQKrrJJ3NGZmJeMWR1346CM4/fS0DOxRR+UdjZlZSTlx1IX/+z+YMyfN2VjKP1Iza9r8LvdDjRkDt92WWhwbbZR3NGZmJVfSxCFpV0mvS5omaUAl2ztLGiXpJUljJHXKHv+5pBcLvr6W9OtsmyRdJOkNSVMk/a6Ur6Fa8+alAfGuXeHcc3MLw8ysPpVscFxSC2AwsBMwA5ggaVhEvFqw2+XA7RExVFIfYBBwaESMBjbJzrMSMA14IjvmCGANYP2IWCgpv5HoSy+F11+H4cOhbdvcwjAzq0+lbHH0BKZFxFsR8Q1wL7BnhX26AaOy26Mr2Q6wLzA8IuZm948HBkbEQoCImFXnkRdj6lS46CLo2xd23TWXEMzM8lDKxLE68G7B/RnZY4UmAftkt/cClpO0coV9DgDuKbi/NrC/pDJJwyVVuoC3pGOzfcpmz569xC+iUhGprEirVnD11XV7bjOzBq6UiaOykrBR4f7pQG9JLwC9gfeA+d+dQFoV+CkwsuCYVsDXEdEDuBG4pbInj4ghEdEjInp07NhxyV9FZe6+G0aNgkGDYNVV6/bcZmYNXCknAM4gjUWU6wTMLNwhImYCewNIagfsExFzCnbpCzwcEd9WOO+D2e2HgVvrOO7qffIJnHIKbLEFHHdcvT61mVlDUMoWxwRgXUldJS1D6nIaVriDpA6SymM4i++3Hg5k8W4qgEeAPtnt3sAbdRp1TQYMSMnjhhugRYt6fWozs4agZIkjIuYD/UndTFOAv0XEK5IGSvpVttv2wOuS3gB+BFxUfrykLqQWy9gKp74E2EfSy6SrsI4u1Wv4nvHj4cYb4eSToXv3entaM7OGRBEVhx2anh49ekRZWdkPO8k338Cmm8IXX8Arr6RihmZmTZikidl48mJc5LBYV16ZEsawYU4aZtasueRIMd56C84/H/beG/bYI+9ozMxy5cRRkwjo1w9atoRrrsk7GjOz3Lmrqib33w8jRqSJfp065R2NmVnu3OKozmefwUknwWabQf/+eUdjZtYguMVRnXPOgVmz4LHHPGfDzCzjFkd1unaFM85ILQ4zMwPc4qje6afnHYGZWYPjFoeZmdWKE4eZmdWKE4eZmdWKE4eZmdWKE4eZmdWKE4eZmdWKE4eZmdWKE4eZmdVKs1jISdJsYPoSHt4B+KgOw6krjqt2HFftOK7aaapxdY6IjhUfbBaJ44eQVFbZClh5c1y147hqx3HVTnOLy11VZmZWK04cZmZWK04cNRuSdwBVcFy147hqx3HVTrOKy2McZmZWK25xmJlZrThxmJlZrThxVEHSGpJGS5oi6RVJJ+UdE4Ck1pL+I2lSFtf5ecdUTlILSS9IeizvWApJ+q+klyW9KKks73jKSVpB0gOSXsv+zrZqADGtl/2cyr8+l3Ry3nEBSDol+5ufLOkeSa3zjglA0klZTK/k+bOSdIukWZImFzy2kqQnJU3Nvq9YF8/lxFG1+cBpEbEBsCXQT1K3nGMCmAf0iYjuwCbArpK2zDmmcicBU/IOogo/j4hNGti19tcAIyJifaA7DeBnFxGvZz+nTYDNgLnAwzmHhaTVgd8BPSJiI6AFcEC+UYGkjYBjgJ6k3+HuktbNKZzbgF0rPDYAGBUR6wKjsvs/mBNHFSLi/Yh4Prv9BemfevV8o4JIvszuLp195X6Fg6ROwC+Bm/KOpTGQ1B7YDrgZICK+iYjP8o3qe3YA3oyIJa26UNdaAm0ktQTaAjNzjgdgA+DZiJgbEfOBscBeeQQSEU8Dn1R4eE9gaHZ7KPDrunguJ44iSOoC/Ax4Lt9IkqxL6EVgFvBkRDSEuK4GzgAW5h1IJQJ4QtJEScfmHUxmLWA2cGvWvXeTpGXzDqqCA4B78g4CICLeAy4H3gHeB+ZExBP5RgXAZGA7SStLagv8Algj55gK/Sgi3of0YRhYpS5O6sRRA0ntgAeBkyPi87zjAYiIBVlXQiegZ9Zczo2k3YFZETExzziq0SsiNgV2I3U5bpd3QKRPz5sC10XEz4CvqKNuhLogaRngV8D9eccCkPXN7wl0BVYDlpV0SL5RQURMAf4EPAmMACaRurmbNCeOakhampQ07oqIh/KOp6Ksa2MM3+/XrG+9gF9J+i9wL9BH0p35hrRIRMzMvs8i9df3zDciAGYAMwpaiw+QEklDsRvwfER8mHcgmR2BtyNidkR8CzwEbJ1zTABExM0RsWlEbEfqKpqad0wFPpS0KkD2fVZdnNSJowqSROp/nhIRV+YdTzlJHSWtkN1uQ/qHei3PmCLirIjoFBFdSN0b/4yI3D8NAkhaVtJy5beBnUndC7mKiA+AdyWtlz20A/BqjiFVdCANpJsq8w6wpaS22f/mDjSAiwkAJK2SfV8T2JuG9XMbBhye3T4c+HtdnLRlXZykieoFHAq8nI0nAJwdEY/nGBPAqsBQSS1Iif9vEdGgLn9tYH4EPJzea2gJ3B0RI/IN6TsnAndl3UJvAUfmHA8AWV/9TsBxecdSLiKek/QA8DypK+gFGk6ZjwclrQx8C/SLiE/zCELSPcD2QAdJM4A/ApcAf5N0FCn57lcnz+WSI2ZmVhvuqjIzs1px4jAzs1px4jAzs1px4jAzs1px4jAzs1px4jCrQ5K2z7M6sKQjJF2b1/Nb8+DEYWbfyeYHmVXLicOaHUmHZGuavCjphvI3S0lfSrpC0vOSRknqmD2+iaRnJb0k6eHyNQ0krSPpqWxtlOclrZ09RbuCdTbuymY6V4xhjKQ/ZXG8IWnb7PHFWgySHpO0fUF8f8qKNT4lqWd2nrck/arg9GtIGiHpdUl/LPJ1D5T0HJD7miDW8DlxWLMiaQNgf1Lhw02ABcDB2eZlSfWZNiWVxy5/070dODMiNgZeLnj8LmBwtjbK1qSqrZAqKZ8MdCNVwe1VRTgtI6Jntu8fq9in0LLAmIjYDPgCuJA0w3svYGDBfj2z17QJsJ+kHkW87skRsUVEjCsiDmvmXHLEmpsdSAsUTcgaAm1YVPhtIXBfdvtO4CFJywMrRMTY7PGhwP1Z/avVI+JhgIj4GiA7538iYkZ2/0WgC1DZG3J54cyJ2T41+YZUgRVSApsXEd9KernC8U9GxMfZ8z8EbEMq01HV615AKuZpVhQnDmtuBAyNiLOK2Le6ejzf634qMK/g9gKq/j+bV8k+81m8J6BwedRvY1GNoIXlx0fEwmxxo6riDqp/3V9HxIIqYjT7HndVWXMzCti3oKLpSpI6Z9uWAvbNbh8EjIuIOcCn5WMQpMKXY7O1WWZI+nV2nlZZccAf6r/AJpKWkrQGS1YCfqfsdbUhrfg2nupft1mtuMVhzUpEvCrp96QVAZciq2gKTCctprShpInAHNKYAKRy1NdniaGwiu2hwA2SBmbnqYvKo+OBt0ldUZNJ1WBraxxwB7AOqRpwGUA1r9usVlwd1ywj6cuIaJd3HGYNnbuqzMysVtziMDOzWnGLw8zMasWJw8zMasWJw8zMasWJw8zMasWJw8zMauX/AWFVESTQvY98AAAAAElFTkSuQmCC\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], + "source": [ + "def eval_show(epoch_per_eval):\n", + " plt.xlabel(\"epoch number\")\n", + " plt.ylabel(\"Model accuracy\")\n", + " plt.title(\"Model accuracy variation chart\")\n", + " plt.plot(epoch_per_eval[\"epoch\"],epoch_per_eval[\"acc\"],\"red\")\n", + " plt.show()\n", + " \n", + "eval_show(epoch_per_eval)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "从上图可以一目了然地挑选出需要的最优模型。" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 总结" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "本例使用MNIST数据集通过卷积神经网络LeNet5进行训练，着重介绍了利用回调函数在进行模型训练的同时进行模型的验证，保存对应`epoch`的模型，并从中挑选出最优模型的方法。" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.6" + }, + "toc": { + "base_numbering": 1, + "nav_menu": {}, + "number_sections": true, + "sideBar": true, + "skip_h1_title": false, + "title_cell": "Table of Contents", + "title_sidebar": "Contents", + "toc_cell": false, + "toc_position": {}, + "toc_section_display": true, + "toc_window_display": true + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/tutorials/source_zh_cn/advanced_use/images/synchronization_training_and_evaluation.png b/tutorials/source_zh_cn/advanced_use/images/synchronization_training_and_evaluation.png new file mode 100644 index 0000000000000000000000000000000000000000..cbecb6c9739eaf047c89ea79f9d596a2793e6283 GIT binary patch literal 12868 zcmaib2UJsGux20>r5dVq4NXCc(z{9x2q?WsQ3MRVN{1jSDAFRmssxZ;LJ1uNM5z*b z4H3RWYf}IUL1un|^sGIp1x;yyz zJ@K*!xjymnaB=r>fj_GzuVfa#zgcjRGnlHa8-zf8BN zk4(obCpTWlBywE(tOa!vdKdLB6$A1*JICZCEO;cRb@TX=zDaZ#(+>dKNpMI5^mQE{aQ_t7JV>)6C5?WlPri z{w6RX8xkva+L7Gu(U|qus55>g%$4qW!L^Ut-$uS}B%NM#{aNUIvY&vrfuZ^SKHoi^ z*o>>!S57c@R_~mLK5aBONb*TXZE$e5+Nsf$>iM|x5?UKI<~)6v^f<>-#}VPLDUoAf zmR_;Ush`}fBz4`X!i975VKuUzW_D%ZP$T@&HQdx97oQ04ngwi_uUT}B36@pmn#G?b zxTXXvCsmO0+B8PyYX$0H(~$Bh45=tfzGhzd8gUSB*OgX*esoln5K z!+5j<4YBQ$`@H;I7Cy1Z4Lw!vMVi`yZ2odV+n+Z%d0^}!YmMpzzhrzE(mDa(>Ov?& ziYEDlMjZa3hefal#DyjJSVydGC;B{oanbpCzx4*Zj$-dZ2TP7r2I)EfQj%hCMR6f6 zzy}HVO4KU*`FlR!p3w1foj@6aO2vcP(w|*E@hF+9`HLU@r=l6BA@&FEn~F&31iaco zVLkJJR?s$LlhX>Z6z+#` zo2<1V9Rxd*kV=0)m?iV)_?@^l4n86?t4uh=s2@m}9{9`Q3~iS;_pIA`P-3A3kVJUW zyTu$ZXI5988ShSE{gFPn%>Bw>ic0S0s>J+@U3&@HZt3X)U9aH*muHCaqf&4eOujB!)-CoOwvr+&G<9Oiy!AlC(puu z<$%xL#S+-W?=)VB@9uwgowo0OL1NMuYgO``aMxDtMOX`cJ*~8H7kxw+cD8+sfZ= z7Hf&-gmX7Q4jmMPhj0Y+uY}t0=Lk+qe>pmw zp4g!gjD0sZV{ETmPd5>V|K#l&;i`lvYuivCY}PwjxwJODn?YO4j28&m$U*u_1#3Bl zl>9WfYcBJ12kn*~rv)p!qEG*kV^^CuvTUkzKTjxBOiSJx`>G^pS=C@sh(_S~cFh}P z%hrdpAS*is^%GQMLcn<10r=bmf1+Z*Uh?qhxYQDVLftw2kb1A~nLW#CGi%EgrY1w5 z3r2%p)F*)1{%lR`IR9*xuX!lhX<5Qi#u?khSI@-?4fW0qRP?WTL0?U#pg6n;+Hc6a zn8e-wupnOfZgd@U6U@mI9~EVfp2k8d=|raKCti<0E#Bb!nXoeWL&PAWT*_-ELsyZ$ zFvD^0bGHrrS&mpi{6HDyDfoe(&#_fdAiPyYdVt3j;97U>pXt+QAJ}M73 zZa-qnqff9>2hb%`RZI6gtxI2{leMkC%^aEBqudwE8zPQ$RnRaHgY}XKvT!Mo`(Kf@ z!D1)7#>!#r5AQ4LBp?R}D0;#?bC#c|hm^P)w#R(q*EEuaKg4og13QOo&{`LTcWnNw zY}-b%n6j4zXkEVDN0Lus#dpSH*6^H=)W(J!GpcGqZJWRU5bmfY64R>LS)kjjrS|@Z z^^!I&JnBe%x|(cF)sr28;`v~~ndfx#yznlMJ(!FAHjS(ab{(0_cSh&-E=tb6^Cxzv zIPq{7vRjF&3GdL6br4!>cyEkgjFh?wwSF7*NY_O<9q)Q(jPt}%U@1vZ?w$0YL6X5L z+vsIYR9P7->};csR5sA)s60V~#4ozx?MhDihGDoi-qXG@iNC^K=F0ij8#Q)g-syvgTuEF;DXVA~S zsme=7y*9X`rG;Uat>Or}9+`akRe+DtGQYX-tQRkNV-Y){tkSbgu}UoY$&Ci3#z?@; zj^Qk!y%o%J{!Oe~d8E`^@jP1w=@9;W>EK0G&; z^zxWBN9aBW=GXzVQP#v;YDuJ1;0~gZo&c3Kv*^i@pP%g{-g}5r`95w%dM;*}XsEZ|6^N#q8^07OBj< zeqdvH6t9}taD|=>2Xy$9>H?92`E-UNTkW(cZRxC=g8u^fM8N8wfyzz>kK^S+agSZx zdM7-_Yf&6sE9GL=fI0j1q42fmYF5Qps#1k<>1hWP1{y+q`61IM&ukb<1SV@rp9ZW$h5QgQ02})Nr*P#}W-(c)F3r&Y= zz4>BNB=2E)@{TY3U=2E0>e2V}eL~NUc1L{5r1gVO3|1_`IrN;0gebqqwQqC(v)MAW8^KDX>Q0LZnt%h zzMJh6`ZcUS?EBue7RK6~<4>xm%+_G!pWdW&_{!=Z<2GVHKXpkWZpU7oS%1raFiGG(SF>K&7LP3I|CRdUL z*%`rfwT2Ia#~LkpPCUuJSJzXNCTl`6cPmg)E~sq6yN!GDY!;En=NlI#sP*=l_KTov zP=RNJERsz6VC+n^j{F1c4OnizZbq%gPG_I?VHazmz>?*>_PEq`tBNnbNQ=L(jbD;6 zR{alz*C70(v91)Ork!aMdnm0e$<=FcnnK}q5!+al6r$y>m_gN~HkJh`O*R%4HEkKV z7ll!To@lCFWGe-Uj&R&YGc+HtyVTnkkPR82+0JnkEFxgW| zHNOeP_SI~Z(=UC?tIv09xZL39J}oQntF0QWa-rxOXKrjH;i(8Nxe51$S%rs$ALpYJ zM^p{2zxjYDMiboQEudC>BKMtQ9z;c{Xm<8MhA5Bd#BW1uOBRE~(a8@ok{znfe&`!8 zA7MN_aN*P{eYG`Xj~Vq8Xx&=5&d(2npZ>-bCCfse(s?lLg2YKMa?o!&?6(RZLL2yZ zk*K$A@7l~K_v=W@L4zO4A{1F67hgJX zMm}GEP$8X8^ATk({mJ1DPp(Exj{*%k)^4TGhigGB4EA>EWW!L#ScFYNK=+OR&wDLq zbJ|D^%}!F)g~xJk4$A@Sg?QKVs$PRX{wHfwI_&XJ6&J`bC+kLt=gdn$Sf7`sq+q2m z5xnoScf1g3cxY7T7N9v+ZK$uyydKF#kt3<_a=r%#5Q=2TLR$8@WY6%D<0BYP_=Ibz zrAwt`&-$q8LS1V=z+?xIE2Og|l-apdKT2;oUC9`JIBlQ&L!k=mT5oTGUV2I0HZCvx z6`*2298jnj)`Pq5>;wxk?{C$Z&ONFbkXvngC!Ehr;W#*;kp5y2pkwhNY*lPGz)xbM z#q@^Xlwdu+eSGn3(tbfdOxwx@zyd$Xt!&0fVdl>s`GC8M`-?Ymf=0L|etIZreD2RU@BIXTL<=ckSnKEsmf%ZY}z zr$XamKAt1cBJG+gPS?ic4WCCxA99vPcFsp+<2KQTDcvU<1%&eEWIxo{&k}RE!B-OSM{h^}Q%~8Y_mb)Xn)^xRC^6YMmF)G0 zCC0$|P`e+vn1%esZ{c$;GQ*}v3gLVM6;I$09)x?Cg2?rT!@0GWdCQKc{<1hkKrhDH zWP^{%rb6`#IgDkIQjPamIU7MwQyJxou#Ut zm}eF%UNa`FfKQH=@LQ#ZZ|Bv&#p##EKj|>=G#M}3J=-Zr{!SnI@Ju~yee%~2kKJ}S zdo0hnIRE=Vmt$Hsq<|`?G|j=R%+7Nsq$Q^b7vN=bH4A+6badH48zo@J?6k4SP(U;L zcYX*y$qB(mPJ~nDr4h)QAZPl^~jFqF93sc#7KE*6Z zXDO@t2Gtcvr&!_JtyA$spm6%xpI>+p|M`%G?MB1BNp0ag)SutE+~jESe0_bx*PC@V z4J^i({2|Dj$y$@u0gvURHAnBwN}?J7E4iVYp4d{!9qMo2kSy=_rnFXj{?&YdU{=ty z>o5~5m;&X?ttU&89Wr)?UB0TUSaZj%p1)V)`TpkQ?%(o)jq6~_uNKb6J2lg2eU;Z2gw~OCOXt#R z2aOBzhT<*splt{3aI09;Lmn`h_Ah5n6Z7GM0*jb%aB+qtDA^3!JYeOQm;8M>rC?2r zf?jV{bpD7U(~rC9*ui({x&F8JFxcW(YQyD2^q@RkGS9MZ#QyKeM&a&|#jnmMrQTi4 z@Q{ewZ4T?uW69yQ4*CLTsOp=$DWbc*Hu-CMwG&X|verZE>2JouEk~rG7ePm3bp~Fo z7v}+^*s_l;-hjH4*JL2&hn2C3TtD;aa-P3+ zNu}MeI-b?$0S`*JBwks|y;bS2ou4FRcf2(}Apcv8PQ+!mh99up$2wwJW87>i%FIO; z9{@G|qlRkvqpu7Lvt1r>tFDQX?Eg{mWG%vHsv5sgLD(+juNpNe%qUjMkfseiKqoR$ z(r7U<9c*X$8Es966531a@;B5rNu%UhCNl zcDH)Mw#T$<#o56cY0wk-#DgR9^|_#7@#&y#nDYY~o^8~l%dA$3Ew-G2&l?<`QGeNM z4K9{reO%!I2x$$=Z?%tENtfJPbgj#yUNg16oZ+5bHks*Tu)bdSm+y|ZLs8ZwykLa( zuur1Uxy*aDiFe~_;K763q3mD=(Pl}a6){F8m0^j4jnTfC7O^T$lb5n${5>uf;yiB~ zAxV3h=J-{({1?~S%j`g*(u)vR#3BD40AP}h>w5~7!ebk9UPcEAhr#)ejN3KqTt zfQ$TSZ2jck9j)Shr9DE$eHnDi z5%g2iWp5b`v)b1R3UV$mo)yc$4o8hmAh2nWut)dm)cPPn#evxhe~tb_57)*0$o(yLJ$f5!_$ijUo7I!$W)YNKTSETMXH%Xd6)p;=}=Q^W3TO@D1(YWzWNwRLdinz!PC89SAt z&-0TPgq<)2_z1Gh%CLXX(mgLQM}AeG)Oqe}*{-QOi%sxclbK;bzB8lEnG<^!dO-fB z-01@Sz7yD?UzjOEKuJ(5vry5|g3K75MjM^PwKA zbH7V=1f903w`Sb-Ps0{W*OTSKx|b?wFB&$|UXc_h*Ok5>3hC0TolmIKrG!O3)%Qzt zolGa17yIsHJGp4YXj?V-etX-&;?pYvKclLX?n|#doZF4yL46u4Z%Ox)bnSWDvBH{= zzTjL=tFRgGC+SdAEFR&OBI+v>;bzPjiQ?jG=7S?qj2x45mRJwi5?@T>Nb&P&IpjB; z^lNKz+AJwPaj1)D(?F@@Xah-o-lsiYY+BTNys9$Xu;csLDTf9r4^AOfAU#8bBTu9z zwiIzwQD!_>X)?388rdr3DwHckcXe&=E~G}8aZFsAc#Mr54W#$|K_2F?d^|abYn<2h zu5aGgn;WWr>lB(dLzc<}Dc9({h10N2p&;PFZyXObnOuP?&`)MFCNr{Bc1Vs|=Ox5u z<{)WvMGugpVm3r&7cZH)R5s<_cIE-XVPDyt&~zu?^}Eq;Re`u)Q`@4f&%2zo$sF(R zb?~@y6`@o`Xe7~rt#d>4;~9P=ZXVoKd+_|qcvTyHT8cmWw-7TH!WZ~(eiBPRQ+>P(6wkx*5=8+NSGkKCoUH@88dCM?ZHrBkDLZ z30qMUN>FK1y@BkEdrl#WY!DvJ&hEC5wtV$SYq};*2y5*TU8$t#k%Xme^ev>r&n$W7 z(7(k^xrIf@GEXfF9$6k))+2=qL7>v&ArYzcpVKpGGhN$=hF5P;+-KLo+p34O}bT$MF3)(-!`ha+ZEHZyR#F}O_#QwDQM|L>) zzg9u_1RXt@2;2aIgGiNMT)M>~eSDFT4!o&Uf)=71qE)f~Sgsp2fs=Ob59){6a6P!0 z!=#|mnW9PteJa0uw*LB=YY#B3QP1uApJ+$zYiP$+E?xB`q^!J-MfL7o>{HZdMdzKz z05?j#B9tkQktS4}cldUm3l$}HdMb%O#$HE;k}8tXl)9OX2@uJwf;LGSwv;S2_r(3wNHoMttPGENamMW%o-SeKPf!oNqw@%JO~uTgPRP}yRnIU2ie|Bp73 zlXRY7ou2%wwC+!=!>1(dB8Kehr6!tN6r5C#ANPMO=>gaqt3{3UrKF@S zlXlE(-wJixj{=b?@aW#&%I>en6Wv6YWgaR-@5S{}dze zq|s!NS}l=SYOz<2*#HVxRb#j{H6I>sc=n}}7njqWozzS#8>8vqM3*+%tni+l+{$^Q z=x*`&QoQ7SYzM9T=;h&}uYfw$z|-t;gJEx0b0WQ`2AL3fB_NFFP=_1p6UH0*dA8)6 z^|(mvSE{g_P%5z|+!QU^qIyu-qnis(bgK7|`TO&9D|As7XKDNvD>O(P zb|)_E6@T(`8HwM?SmT=Dq0&gy772YXT#TrB1e3=QQ4Hw(Z~2I?@e;6^HTe7*h{PK) zj!-RPOElK{gxg6pBoc0f;(0L(6ta72M+@{%YDWLWeu0Sy#73Dli;z|d5%rUBz1=>g zU48>K$+g|0pkx}My;dY|fUQ!GOZo6i|Fgv^;IR%0^m$}AZeb3suS(zAd7f_6Gm6d8 z=!|?#3AxR61pj-#>fOJgeDFg+VzV1<(-yR|%%z*F4dcv01#IQHEOty48B`uKn;cEQ zurzaT2Jjs1<+Gic&wISvk$$#ls=WwQXU(x=$Xe1jB!{Llw=8b}p#s&p6J8}Nt;V+b zFAntqce1f?K1INY*sv?WdU)oRKB(<~xX<0a)zKfcWfF6L#$XH5!5GL2-l&S2#VF0aaw)7bSaAhUKW%D;5kecmJ+f<4fV?`tZeevbYaHY`XzMS%yR9`W{Ptrz+-_s{LDvp}IiV$ro z#^+oE#ER|n5KZEExks}*A5F!Gv08{R7?6Sle@O1`wG903Fn~g(D}YF29Sidep5R_x zzWH_pVUW+sMm7Uo+uGdIZc`qm{$T-xpIG*9gq@s~=G#Eoq?>Da{$SFv!#b$FVPl0| z?|S_zqih-mJD9 zShzhb)SPO$q6LPKd>5F4^inriGXr^|UQF~$YhE+@)nEPijrqUSUabnEteSrLZPrh;M%jz$~ z=(b_wm>zAvc5;BzVi`@&gm>ULC6>96x_+CM#7D-OYMqggJyu&w+{cpMot_aos@%~b3m4~XKk=_NNc1d4Er|5Wk&>#L3tuIwAYQRw>V)-{JQb@ z?yn=;NT3QA{9-^Xr(B2lKpUVUYyZi;;bush%3xtL|$5>PP3KAijihCeU*a$xAc zWoff7zKjKZfdjBgCrfljf7i!b^|}y|JaiHc)swv@-X|lv2pD9~Nv)66jqC-yLSF^T z_YMSXwNd_zskR;Y+y-qkf(eu8Xm$cMjEGj2iJEsipI%tKU<*EhH{5V@_*>{~idh#* zy+f||_@iNgQ~Co=WU~9V$=7<*|EXZwt?#yw`TB_dgQPb0UjTH)P2rBSNxH}|z^I9t z!vRo}>lLaFUs)+)*F9O8pOAX>art_^E|=*h8IA!)363LmPk3`yMuI1&8du~-R+Go* zmreY>yAp-r#BgHo>xpzuwJ=egmo*6OX!Vt_WO`DhKjaDlfgs7mUVs@-%2Ry7?u_@( zQcGi-vL7s(Y9-tdoBnfGf8o4XyE_$5Lrd0e(_IQ`ozsxYAT<}V%uFSjh>;WjJ5c3; ze9u<;2}uRV@|A(f1;7jRv0FTSD`_*^`BvQDW7MWB@unu<8VJ_mAqYIjApxb@`&bizeRN zXbbQ2G%8wGK-que|}xB$EU;rnU9wg*f6+{^@dd58~t2VaehYV{_-gI51rkD zZssNDQu5X_k<8FQqSc8s96L-n^2i=OmR)%DT0RLi1Q}{w*w3q$b#RnoOUU{nDDL;^ zbLCk5@6VE_1HdwmB{mg+_FTPnoq=WCPgMr+vLW z=RDCNNp`cMsPCgCfDmEeQ1znqY|>43NS%=)gR}L2V1&7ji8;R&ACDYEcG+oWwXXHj z^TO|9P~|WFlE&EWa-t&*IYAYYa|es>Jnt(}N>yO}7H9JXx{zPynP>sRu!}1-w%&qc zHrGpIzx8+z>%z6ei_6^tY$PPKBz~K~9p17~gJ2!-u&WRy=x2ppFs`;p&5rt=ma`j? zPbamTtlh0_kV6)3wa$>fjKZ;^exfehrEe4o`RikEw*5tw6@w9n#q3G_63;H-H`iXO zNEEXH@}VGAf~k7#fK0w;qn(6HcKwbPtQUY0HKIjgUGO7fd?K`c{Q_LowH^zviTaRk z*uxjO0b^H9mvL%B4I#%$L!bL@N6_QS!+hIw(nd248j3{0vC&a;kRcA)Y?mzRVkqD; z%wqr_aSH6=iFbH>$cRLFL9`eBQ*- zsthbRRj$VrHN;3(6$$PxIZ+soH+=jbtz-v5oR<7?ei0r9$n1;xEp!481G|h zI1Pe6U0tcQ6ee1T(_?RuSa+~HX$PV}i;>epika$MPid4WAauG8rwvIhzwz50${8?C zbkv?J#LTK;_dXD%8r^hrFv(5N;5v+s!l*#$NTtaINPwo2S%;y~4~O&mrnE)3x`htpP|=)*yTLGt`aU9r|DDhsRnP9SC(Cwq7x6X8-#V&!}llMIJ_6 z(=ag=l_=*aCBfL*MSy1 z!cLoK_cGVpw8{I$4$*duwgzfV><~19BixF8bC~{W_AJlP_ZJAIZ~}Dk!=@+bwY_r+ z$z0j^Eip-1en^$ukIys6)uvWZ$*8Y)0?@@pr0S_>QKU7>?am%TA0N zk^=0sMg5*fI}VPuM!#PE4@TsczyC)@{QPGr6T}_d2-dP-nBavvFl|-Icm%2S4IHdp zcDVx4x1V+=-7eQVVt#nzNMVZ@v6)9c>SyVeXiG<$q>LnuEc=-5(IEr1ggQMKJbWX( zX`Y?C6|J@1LAGQo&9S6D-1{!v1w=mPltwjxcj`;1Q#9`YpK;#Qob?5civt4_L(HlIr2SdiI`omP6)$o!YB%qHIz*d@^K9;tQo z(sAiuoS)f^e0k(b_}oh1$_&2KwENOo22f?Bq&)OQCU`DsX%ygkIy6X-R${}WiCa?& z(AYP0{X~s@=N+Ak1+)^_ewcOxHjW$|m{(ae^J1py`@3dU*^ z%NhayOr^tX@e3bO5L_OMUlI@J{-FywK(a>#z<^+;tkA;#-~v*IXa>IR9iGBO%P`TL zSu`^v>b?9DOZQhVA|kQX$+9F9Ra<_kvr-%=aAH=uiQ`&%E}C6swE`kfiDCsk&QfDQ z^t*Te^Um|q`|a}|@BK6D{7hMk4&vb5ikx*)ZcRhP?{zbaH%!$my#L)OAf5nOOHuJZ zZhnZwdI9?}8jrT>Y=8<)+Va=8xtu^|#~Tae8EK;;g0WseNkBqEFc#T-$o)8|ln4#m za-NCA1m6L{s0jx|rm%@4au%e=nRkQfG(gFdNY4@g{0@DL0aftxYP4ZIftgD5V!DPH z0JhWmyWi%@5n;oHM31u{C|{nPt`a9;B9SSZzCT)?-#|+w3;Ds?ov~kRv&zQC2>j3atyF#AR2vpTT$B- zRYw`*dkni;el$z~lco)5Je>P`;*2uY@!y&MLc~ZB@hb5u7e}}$i zOtGJxzR7>>h#TlUwXtov>A?RNr2M_iG_&(%X=M21e%Cgrw&#Eg^uXn2eAMsBbeZ@r zSs~}y5vFcq?te#iV05vQl`-TtrR=+re#!sZ^@gvy)QN)z7ocwlZUC-r8OCzGNMK8$}+fM1{cCA+0Xivi2%(!y zJW?PrUE$IRx(0{aQsaaWQK?C^Qsw-X;BqxJXGg0q)uL@rkqw~eVOxQBnRxQ-_fh&) zQ#G1e``b13QT|LvAKQl;L8@qiYPgl~o`oJK|Mzzg8Pe+ADl|s7ZpXB zp;DyV1ai4v*of*1-;ua5(h0D<4b8uMa)-v~85B`zI8kEhL*->s~X^9fa0=b}PxL98mgoF!<#5N;`gpd4dhY zzheffYB{z-jD{zL^Yd}VyBl4BFd|ey^dyHL98R-`H}7!)z$XFpfoQXbMMq8li7oh4 z;%vu9ec)VM(;uG2f0k0c-D83*8RQ-^ocCMSzTaO!bE|=weaSk%pSfy>+2Lxt-K^5BeS0#QLPI6aGd6(0C1o zI$s4cDn6n@{!OfQc@cIS%QF)8=RHv2eivm88HyZf-t$oVb?$oQE|vkAEHpz|%j+xl zO9(ngGM89waxgO17TwQ@9iuSdoi$36LFn>}WS_aZ>C(~&fNqrB7lKP+UFK`>-i_3< zHyXpPcsTOwS&Y^fP1W~@gTq(R8<#;BDgCWU1xbiclh~##tq+=GBfI15XsAcfNy#T_ z4^_399Cs}ndG;gAF%i7b?o#Fw$+eB*d6pydxk@IrPD7*{c{Y$S#(a8_e|!{F3z~Jo zeUD1b*1UpzlOPbgbsc`kLkMm#k-^V(U%Ms9_*aeMaui_S9sg6`UGX)T<%*m_1uBBy z3OT-E8A9v<6SMz&&!OLIv7&)@4dJ5up%N)79Bih2Vun3NW+zELHW=JIrsY?<3C~1L zQ#-S3S*T)F&^3>;lkhd0{K>a09GBe9^SnE*YG|LRvUID#=KHgA%Db+uxxp-1x(*{< zs=kG3Pw^=X79(GX%AXY>w^Q&)l%Y=Wd<4G|`L7m%s8MA>x?*AE2`M37KzZiM5$S}= zbQ2joG~Ycwzgsl;w>9;tJamjQTP*P0y)AEY>PAL_gX?R_4}Y$GFyn#QAHRB*TF0(%%q{*KeBnKW{~}1HQH)3bXOobB+piZ<)H$O zaowaLh;PdA)Qpa?0|)RQx3DxN@eC8IklU8necITt8_>By>kDa|0tzml9yo6kFR(8u z{FX6tl!HW-cM-*K;OeHFP2Q06F4)@Vf;#O>@cIwLi!}$7y+zAcw<4Fn3%OTO0nX)Fo_Z zO58JN^n>JT=NH-P=pQ7kV`1U6{JzL)q0$V}(CN((5e8 z6f=oJLjQBBCOwM>a4uMGI2+2DG(JD~TqpVJJI6|;E0_mZ+@@kuY5?A*%^6kpg;fjP zp1k3SleeYCy0qnj(gWrl?DsY?4s0-BTL>A@{lsJg=*A|xlmCcE{ + +- [同步训练和验证模型](#同步训练和验证模型) + - [概述](#概述) + - [定义回调函数EvalCallBack](#定义回调函数evalcallback) + - [定义训练网络并执行](#定义训练网络并执行) + - [定义函数绘制不同epoch下模型的精度](#定义函数绘制不同epoch下模型的精度) + - [总结](#总结) + + + + +   + + +## 概述 + +在面对复杂网络时，往往需要进行几十甚至几百次的epoch训练。在训练之前，很难掌握在训练到第几个epoch时，模型的精度能达到满足要求的程度，所以经常会采用一边训练的同时，在相隔固定epoch的位置对模型进行精度验证，并保存相应的模型，等训练完毕后，通过查看对应模型精度的变化就能迅速地挑选出相对最优的模型，本文将采用这种方法，以LeNet网络为样本，进行示例。 + +流程如下： +1. 定义回调函数EvalCallBack，实现同步进行训练和验证。 +2. 定义训练网络并执行。 +3. 将不同epoch下的模型精度绘制出折线图并挑选最优模型。 + +完整示例请参考[notebook](https://gitee.com/mindspore/docs/blob/master/tutorials/notebook/synchronization_training_and_evaluation.ipynb)。 + +## 定义回调函数EvalCallBack + +实现思想：每隔n个epoch验证一次模型精度，由于在自定义函数中实现，如需了解详细用法，请参考[API说明](https://www.mindspore.cn/api/zh-CN/master/api/python/mindspore/mindspore.train.html?highlight=callback#mindspore.train.callback.Callback)； + +核心实现：回调函数的`epoch_end`内设置验证点，如下： + +`cur_epoch % eval_per_epoch == 0`：即每`eval_per_epoch`个epoch结束时，验证一次模型精度。 + +- `cur_epoch`：当前训练过程的epoch数值。 +- `eval_per_epoch`：用户自定义数值，即验证频次。 + +其他参数解释： + +- `model`：即是MindSpore中的`Model`函数。 +- `eval_dataset`：验证数据集。 +- `epoch_per_eval`：记录验证模型的精度和相应的epoch数，其数据形式为`{"epoch":[],"acc":[]}`。 + +```python +import matplotlib.pyplot as plt +from mindspore.train.callback import Callback + +class EvalCallBack(Callback): + def __init__(self, model, eval_dataset, eval_per_epoch): + self.model = model + self.eval_dataset = eval_dataset + self.eval_per_epoch = eval_per_epoch + + def epoch_end(self, run_context): + cb_param = run_context.original_args() + cur_epoch = cb_param.cur_epoch_num + if cur_epoch % self.eval_per_epoch == 0: + acc = self.model.eval(self.eval_dataset,dataset_sink_mode = True) + epoch_per_eval["epoch"].append(cur_epoch) + epoch_per_eval["acc"].append(acc["Accuracy"]) + print(acc) + +``` + +## 定义训练网络并执行 + +在保存模型的参数`CheckpointConfig`中，需计算好单个epoch中的step数，再根据需要进行验证模型精度的频次对应，本次示例为1875个step/epoch，按照每两个epoch验证一次的思想，这里设置`save_checkpoint_steps=eval_per_epoch*1875`，其中变量`eval_per_epoch`等于2。 + +参数解释： + +- `config_ck`：定义保存模型信息。 + - `save_checkpoint_steps`：每多少个step保存一次模型。 + - `keep_checkpoint_max`：设置保存模型数量的上限。 +- `ckpoint_cb`：定义模型保存的名称及路径信息。 +- `model`：定义模型。 +- `model.train`：模型训练函数。 +- `epoch_per_eval`：定义收集`epoch`数和对应模型精度信息的字典。 + +```python +from mindspore.train.serialization import load_checkpoint, load_param_into_net +from mindspore.train.callback import ModelCheckpoint, CheckpointConfig, LossMonitor +from mindspore.train import Model +from mindspore import context +from mindspore.nn.metrics import Accuracy +from mindspore.nn.loss import SoftmaxCrossEntropyWithLogits + +if __name__ == "__main__": + context.set_context(mode=context.GRAPH_MODE, device_target="GPU") + ckpt_save_dir = "./lenet_ckpt" + eval_per_epoch = 2 + + ... ... + + # need to calculate how many steps are in each epoch，in this example, 1875 steps per epoch + config_ck = CheckpointConfig(save_checkpoint_steps=eval_per_epoch*1875, keep_checkpoint_max=15) + ckpoint_cb = ModelCheckpoint(prefix="checkpoint_lenet",directory=ckpt_save_dir, config=config_ck) + model = Model(network, net_loss, net_opt, metrics={"Accuracy": Accuracy()}) + + epoch_per_eval = {"epoch":[],"acc":[]} + eval_cb = EvalCallBack(model,eval_data,eval_per_epoch) + + model.train(epoch_size, train_data, callbacks=[ckpoint_cb, LossMonitor(375),eval_cb], + dataset_sink_mode=True) +``` + +输出结果： + + epoch: 1 step: 375, loss is 2.298612 + epoch: 1 step: 750, loss is 2.075152 + epoch: 1 step: 1125, loss is 0.39205977 + epoch: 1 step: 1500, loss is 0.12368304 + epoch: 1 step: 1875, loss is 0.20988345 + epoch: 2 step: 375, loss is 0.20582482 + epoch: 2 step: 750, loss is 0.029070046 + epoch: 2 step: 1125, loss is 0.041760832 + epoch: 2 step: 1500, loss is 0.067035824 + epoch: 2 step: 1875, loss is 0.0050643035 + {'Accuracy': 0.9763621794871795} + + ... ... + + epoch: 9 step: 375, loss is 0.021227183 + epoch: 9 step: 750, loss is 0.005586236 + epoch: 9 step: 1125, loss is 0.029125651 + epoch: 9 step: 1500, loss is 0.00045874066 + epoch: 9 step: 1875, loss is 0.023556218 + epoch: 10 step: 375, loss is 0.0005807788 + epoch: 10 step: 750, loss is 0.02574059 + epoch: 10 step: 1125, loss is 0.108463734 + epoch: 10 step: 1500, loss is 0.01950589 + epoch: 10 step: 1875, loss is 0.10563098 + {'Accuracy': 0.979667467948718} + + +在同一目录找到`lenet_ckpt`文件夹，文件夹中保存了5个模型，和一个计算图相关数据，其结构如下： + +``` +lenet_ckpt +├── checkpoint_lenet-10_1875.ckpt +├── checkpoint_lenet-2_1875.ckpt +├── checkpoint_lenet-4_1875.ckpt +├── checkpoint_lenet-6_1875.ckpt +├── checkpoint_lenet-8_1875.ckpt +└── checkpoint_lenet-graph.meta +``` + +## 定义函数绘制不同epoch下模型的精度 + +定义绘图函数`eval_show`，将`epoch_per_eval`载入到`eval_show`中，绘制出不同`epoch`下模型的验证精度折线图。 + + +```python +def eval_show(epoch_per_eval): + plt.xlabel("epoch number") + plt.ylabel("Model accuracy") + plt.title("Model accuracy variation chart") + plt.plot(epoch_per_eval["epoch"],epoch_per_eval["acc"],"red") + plt.show() + +eval_show(epoch_per_eval) +``` + +输出结果： + +![png](./images/synchronization_training_and_evaluation.png) + + +从上图可以一目了然地挑选出需要的最优模型。 + +## 总结 + +本次使用MNIST数据集通过卷积神经网络LeNet5进行训练，着重介绍了在进行模型训练的同时进行模型的验证，保存对应`epoch`的模型，并从中挑选出最优模型的方法。 diff --git a/tutorials/source_zh_cn/index.rst b/tutorials/source_zh_cn/index.rst index 33171ea1..bb6bdfde 100644 --- a/tutorials/source_zh_cn/index.rst +++ b/tutorials/source_zh_cn/index.rst @@ -33,6 +33,7 @@ MindSpore教程 advanced_use/computer_vision_application advanced_use/nlp_application advanced_use/second_order_optimizer_for_resnet50_application + advanced_use/synchronization_training_and_evaluation .. toctree:: :glob: -- GitLab