{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 使用注意力机制的LSTM的机器翻译\n",
    "\n",
    "本示例教程介绍如何使用飞桨完成一个机器翻译任务。我们将会使用飞桨提供的LSTM的API，组建一个`sequence to sequence with attention`的机器翻译的模型，并在示例的数据集上完成从英文翻译成中文的机器翻译。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 环境设置\n",
    "\n",
    "本示例教程基于飞桨2.0-beta版本。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.0.0-beta0\n"
     ]
    }
   ],
   "source": [
    "import paddle\n",
    "import paddle.nn.functional as F\n",
    "import re\n",
    "import numpy as np\n",
    "\n",
    "paddle.disable_static()\n",
    "print(paddle.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 下载数据集\n",
    "\n",
    "我们将使用 [http://www.manythings.org/anki/](http://www.manythings.org/anki/) 提供的中英文的英汉句对作为数据集，来完成本任务。该数据集含有23610个中英文双语的句对。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2020-09-10 16:17:25--  https://www.manythings.org/anki/cmn-eng.zip\n",
      "Resolving www.manythings.org (www.manythings.org)... 2606:4700:3033::6818:6dc4, 2606:4700:3036::ac43:adc6, 2606:4700:3037::6818:6cc4, ...\n",
      "Connecting to www.manythings.org (www.manythings.org)|2606:4700:3033::6818:6dc4|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 1030722 (1007K) [application/zip]\n",
      "Saving to: ‘cmn-eng.zip’\n",
      "\n",
      "cmn-eng.zip         100%[===================>]   1007K  91.2KB/s    in 11s     \n",
      "\n",
      "2020-09-10 16:17:38 (91.2 KB/s) - ‘cmn-eng.zip’ saved [1030722/1030722]\n",
      "\n",
      "Archive:  cmn-eng.zip\n",
      "  inflating: cmn.txt                 \n",
      "  inflating: _about.txt              \n"
     ]
    }
   ],
   "source": [
    "!wget -c https://www.manythings.org/anki/cmn-eng.zip && unzip cmn-eng.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   23610 cmn.txt\n"
     ]
    }
   ],
   "source": [
    "!wc -l cmn.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 构建双语句对的数据结构\n",
    "\n",
    "接下来我们通过处理下载下来的双语句对的文本文件，将双语句对读入到python的数据结构中。这里做了如下的处理。\n",
    "\n",
    "- 对于英文，会把全部英文都变成小写，并只保留英文的单词。\n",
    "- 对于中文，为了简便起见，未做分词，按照字做了切分。\n",
    "- 为了后续的程序运行的更快，我们通过限制句子长度，和只保留部分英文单词开头的句子的方式，得到了一个较小的数据集。这样得到了一个有5508个句对的数据集。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "MAX_LEN = 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5508\n",
      "(['i', 'won'], ['我', '赢', '了', '。'])\n",
      "(['he', 'ran'], ['他', '跑', '了', '。'])\n",
      "(['i', 'quit'], ['我', '退', '出', '。'])\n",
      "(['i', 'm', 'ok'], ['我', '沒', '事', '。'])\n",
      "(['i', 'm', 'up'], ['我', '已', '经', '起', '来', '了', '。'])\n",
      "(['we', 'try'], ['我', '们', '来', '试', '试', '。'])\n",
      "(['he', 'came'], ['他', '来', '了', '。'])\n",
      "(['he', 'runs'], ['他', '跑', '。'])\n",
      "(['i', 'agree'], ['我', '同', '意', '。'])\n",
      "(['i', 'm', 'ill'], ['我', '生', '病', '了', '。'])\n"
     ]
    }
   ],
   "source": [
    "lines = open('cmn.txt', encoding='utf-8').read().strip().split('\\n')\n",
    "words_re = re.compile(r'\\w+')\n",
    "\n",
    "pairs = []\n",
    "for l in lines:\n",
    "    en_sent, cn_sent, _ = l.split('\\t')\n",
    "    pairs.append((words_re.findall(en_sent.lower()), list(cn_sent)))\n",
    "\n",
    "# create a smaller dataset to make the demo process faster\n",
    "filtered_pairs = []\n",
    "\n",
    "for x in pairs:\n",
    "    if len(x[0]) < MAX_LEN and len(x[1]) < MAX_LEN and \\\n",
    "    x[0][0] in ('i', 'you', 'he', 'she', 'we', 'they'):\n",
    "        filtered_pairs.append(x)\n",
    "           \n",
    "print(len(filtered_pairs))\n",
    "for x in filtered_pairs[:10]: print(x) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  创建词表\n",
    "\n",
    "接下来我们分别创建中英文的词表，这两份词表会用来将英文和中文的句子转换为词的ID构成的序列。词表中还加入了如下三个特殊的词：\n",
    "- `<pad>`: 用来对较短的句子进行填充。\n",
    "- `<bos>`: \"begin of sentence\"， 表示句子的开始的特殊词。\n",
    "- `<eos>`: \"end of sentence\"， 表示句子的结束的特殊词。\n",
    "\n",
    "Note: 在实际的任务中，可能还需要通过`<unk>`（或者`<oov>`）特殊词来表示未在词表中出现的词。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2539\n",
      "2039\n"
     ]
    }
   ],
   "source": [
    "en_vocab = {}\n",
    "cn_vocab = {}\n",
    "\n",
    "# create special token for pad, begin of sentence, end of sentence\n",
    "en_vocab['<pad>'], en_vocab['<bos>'], en_vocab['<eos>'] = 0, 1, 2\n",
    "cn_vocab['<pad>'], cn_vocab['<bos>'], cn_vocab['<eos>'] = 0, 1, 2\n",
    "\n",
    "en_idx, cn_idx = 3, 3\n",
    "for en, cn in filtered_pairs:\n",
    "    for w in en: \n",
    "        if w not in en_vocab: \n",
    "            en_vocab[w] = en_idx\n",
    "            en_idx += 1\n",
    "    for w in cn:  \n",
    "        if w not in cn_vocab: \n",
    "            cn_vocab[w] = cn_idx\n",
    "            cn_idx += 1\n",
    "\n",
    "print(len(list(en_vocab)))\n",
    "print(len(list(cn_vocab)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 创建padding过的数据集\n",
    "\n",
    "接下来根据词表，我们将会创建一份实际的用于训练的用numpy array组织起来的数据集。\n",
    "- 所有的句子都通过`<pad>`补充成为了长度相同的句子。\n",
    "- 对于英文句子（源语言），我们将其反转了过来，这会带来更好的翻译的效果。\n",
    "- 所创建的`padded_cn_label_sents`是训练过程中的预测的目标，即，每个中文的当前词去预测下一个词是什么词。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(5508, 11)\n",
      "(5508, 12)\n",
      "(5508, 12)\n"
     ]
    }
   ],
   "source": [
    "padded_en_sents = []\n",
    "padded_cn_sents = []\n",
    "padded_cn_label_sents = []\n",
    "for en, cn in filtered_pairs:\n",
    "    # reverse source sentence\n",
    "    padded_en_sent = en + ['<eos>'] + ['<pad>'] * (MAX_LEN - len(en))\n",
    "    padded_en_sent.reverse()\n",
    "    padded_cn_sent = ['<bos>'] + cn + ['<eos>'] + ['<pad>'] * (MAX_LEN - len(cn))\n",
    "    padded_cn_label_sent = cn + ['<eos>'] + ['<pad>'] * (MAX_LEN - len(cn) + 1) \n",
    "\n",
    "    padded_en_sents.append([en_vocab[w] for w in padded_en_sent])\n",
    "    padded_cn_sents.append([cn_vocab[w] for w in padded_cn_sent])\n",
    "    padded_cn_label_sents.append([cn_vocab[w] for w in padded_cn_label_sent])\n",
    "\n",
    "train_en_sents = np.array(padded_en_sents)\n",
    "train_cn_sents = np.array(padded_cn_sents)\n",
    "train_cn_label_sents = np.array(padded_cn_label_sents)\n",
    "\n",
    "print(train_en_sents.shape)\n",
    "print(train_cn_sents.shape)\n",
    "print(train_cn_label_sents.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 创建网络\n",
    "\n",
    "我们将会创建一个Encoder-AttentionDecoder架构的模型结构用来完成机器翻译任务。\n",
    "首先我们将设置一些必要的网络结构中用到的参数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "embedding_size = 128\n",
    "hidden_size = 256\n",
    "num_encoder_lstm_layers = 1\n",
    "en_vocab_size = len(list(en_vocab))\n",
    "cn_vocab_size = len(list(cn_vocab))\n",
    "epochs = 20\n",
    "batch_size = 16"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Encoder部分\n",
    "\n",
    "在编码器的部分，我们通过查找完Embedding之后接一个LSTM的方式构建一个对源语言编码的网络。飞桨的RNN系列的API，除了LSTM之外，还提供了SimleRNN, GRU供使用，同时，还可以使用反向RNN，双向RNN，多层RNN等形式。也可以通过`dropout`参数设置是否对多层RNN的中间层进行`dropout`处理，来防止过拟合。\n",
    "\n",
    "除了使用序列到序列的RNN操作之外，也可以通过SimpleRNN, GRUCell, LSTMCell等API更灵活的创建单步的RNN计算，甚至通过继承RNNCellBase来实现自己的RNN计算单元。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# encoder: simply learn representation of source sentence\n",
    "class Encoder(paddle.nn.Layer):\n",
    "    def __init__(self):\n",
    "        super(Encoder, self).__init__()\n",
    "        self.emb = paddle.nn.Embedding(en_vocab_size, embedding_size,)\n",
    "        self.lstm = paddle.nn.LSTM(input_size=embedding_size, \n",
    "                                   hidden_size=hidden_size, \n",
    "                                   num_layers=num_encoder_lstm_layers)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = self.emb(x)\n",
    "        x, (_, _) = self.lstm(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## AttentionDecoder部分\n",
    "\n",
    "在解码器部分，我们通过一个带有注意力机制的LSTM来完成解码。\n",
    "\n",
    "- 单步的LSTM：在解码器的实现的部分，我们同样使用LSTM，与Encoder部分不同的是，下面的代码，每次只让LSTM往前计算一次。整体的recurrent部分，是在训练循环内完成的。\n",
    "- 注意力机制：这里使用了一个由两个Linear组成的网络来完成注意力机制的计算，它用来计算出目标语言在每次翻译一个词的时候，需要对源语言当中的每个词需要赋予多少的权重。\n",
    "- 对于第一次接触这样的网络结构来说，下面的代码在理解起来可能稍微有些复杂，你可以通过插入打印每个tensor在不同步骤时的形状的方式来更好的理解。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# only move one step of LSTM, \n",
    "# the recurrent loop is implemented inside training loop\n",
    "class AttentionDecoder(paddle.nn.Layer):\n",
    "    def __init__(self):\n",
    "        super(AttentionDecoder, self).__init__()\n",
    "        self.emb = paddle.nn.Embedding(cn_vocab_size, embedding_size)\n",
    "        self.lstm = paddle.nn.LSTM(input_size=embedding_size + hidden_size, \n",
    "                                   hidden_size=hidden_size)\n",
    "\n",
    "        # for computing attention weights\n",
    "        self.attention_linear1 = paddle.nn.Linear(hidden_size * 2, hidden_size)\n",
    "        self.attention_linear2 = paddle.nn.Linear(hidden_size, 1)\n",
    "        \n",
    "        # for computing output logits\n",
    "        self.outlinear =paddle.nn.Linear(hidden_size, cn_vocab_size)\n",
    "\n",
    "    def forward(self, x, previous_hidden, previous_cell, encoder_outputs):\n",
    "        x = self.emb(x)\n",
    "        \n",
    "        attention_inputs = paddle.concat((encoder_outputs, \n",
    "                                      paddle.tile(previous_hidden, repeat_times=[1, MAX_LEN+1, 1])),\n",
    "                                      axis=-1\n",
    "                                     )\n",
    "\n",
    "        attention_hidden = self.attention_linear1(attention_inputs)\n",
    "        attention_hidden = F.tanh(attention_hidden)\n",
    "        attention_logits = self.attention_linear2(attention_hidden)\n",
    "        attention_logits = paddle.squeeze(attention_logits)\n",
    "\n",
    "        attention_weights = F.softmax(attention_logits)        \n",
    "        attention_weights = paddle.expand_as(paddle.unsqueeze(attention_weights, -1), \n",
    "                                             encoder_outputs)\n",
    "\n",
    "        context_vector = paddle.multiply(encoder_outputs, attention_weights)               \n",
    "        context_vector = paddle.reduce_sum(context_vector, 1)\n",
    "        context_vector = paddle.unsqueeze(context_vector, 1)\n",
    "        \n",
    "        lstm_input = paddle.concat((x, context_vector), axis=-1)\n",
    "\n",
    "        # LSTM requirement to previous hidden/state: \n",
    "        # (number_of_layers * direction, batch, hidden)\n",
    "        previous_hidden = paddle.transpose(previous_hidden, [1, 0, 2])\n",
    "        previous_cell = paddle.transpose(previous_cell, [1, 0, 2])\n",
    "        \n",
    "        x, (hidden, cell) = self.lstm(lstm_input, (previous_hidden, previous_cell))\n",
    "        \n",
    "        # change the return to (batch, number_of_layers * direction, hidden)\n",
    "        hidden = paddle.transpose(hidden, [1, 0, 2])\n",
    "        cell = paddle.transpose(cell, [1, 0, 2])\n",
    "\n",
    "        output = self.outlinear(hidden)\n",
    "        output = paddle.squeeze(output)\n",
    "        return output, (hidden, cell)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 训练模型\n",
    "\n",
    "接下来我们开始训练模型。\n",
    "\n",
    "- 在每个epoch开始之前，我们对训练数据进行了随机打乱。\n",
    "- 我们通过多次调用`atten_decoder`，在这里实现了解码时的recurrent循环。\n",
    "- `teacher forcing`策略: 在每次解码下一个词时，我们给定了训练数据当中的真实词作为了预测下一个词时的输入。相应的，你也可以尝试用模型预测的结果作为下一个词的输入。（或者混合使用）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch:0\n",
      "iter 0, loss:[7.620109]\n",
      "iter 200, loss:[2.9760551]\n",
      "epoch:1\n",
      "iter 0, loss:[2.9679596]\n",
      "iter 200, loss:[3.161064]\n",
      "epoch:2\n",
      "iter 0, loss:[2.7516625]\n",
      "iter 200, loss:[2.9755423]\n",
      "epoch:3\n",
      "iter 0, loss:[2.7249248]\n",
      "iter 200, loss:[2.3419888]\n",
      "epoch:4\n",
      "iter 0, loss:[2.3236473]\n",
      "iter 200, loss:[2.3453429]\n",
      "epoch:5\n",
      "iter 0, loss:[2.1926975]\n",
      "iter 200, loss:[2.1977856]\n",
      "epoch:6\n",
      "iter 0, loss:[2.014393]\n",
      "iter 200, loss:[2.1863418]\n",
      "epoch:7\n",
      "iter 0, loss:[1.8619595]\n",
      "iter 200, loss:[1.8904227]\n",
      "epoch:8\n",
      "iter 0, loss:[1.5901132]\n",
      "iter 200, loss:[1.7812968]\n",
      "epoch:9\n",
      "iter 0, loss:[1.341565]\n",
      "iter 200, loss:[1.4957166]\n",
      "epoch:10\n",
      "iter 0, loss:[1.2202356]\n",
      "iter 200, loss:[1.3485341]\n",
      "epoch:11\n",
      "iter 0, loss:[1.1035374]\n",
      "iter 200, loss:[1.2871654]\n",
      "epoch:12\n",
      "iter 0, loss:[1.194801]\n",
      "iter 200, loss:[1.0479954]\n",
      "epoch:13\n",
      "iter 0, loss:[1.0022258]\n",
      "iter 200, loss:[1.0899843]\n",
      "epoch:14\n",
      "iter 0, loss:[0.93466896]\n",
      "iter 200, loss:[0.99347967]\n",
      "epoch:15\n",
      "iter 0, loss:[0.83665943]\n",
      "iter 200, loss:[0.9594004]\n",
      "epoch:16\n",
      "iter 0, loss:[0.78929776]\n",
      "iter 200, loss:[0.945769]\n",
      "epoch:17\n",
      "iter 0, loss:[0.62574965]\n",
      "iter 200, loss:[0.6308163]\n",
      "epoch:18\n",
      "iter 0, loss:[0.63433456]\n",
      "iter 200, loss:[0.6287957]\n",
      "epoch:19\n",
      "iter 0, loss:[0.54270047]\n",
      "iter 200, loss:[0.72688276]\n"
     ]
    }
   ],
   "source": [
    "encoder = Encoder()\n",
    "atten_decoder = AttentionDecoder()\n",
    "\n",
    "opt = paddle.optimizer.Adam(learning_rate=0.001, \n",
    "                            parameters=encoder.parameters()+atten_decoder.parameters())\n",
    "\n",
    "for epoch in range(epochs):\n",
    "    print(\"epoch:{}\".format(epoch))\n",
    "\n",
    "    # shuffle training data\n",
    "    perm = np.random.permutation(len(train_en_sents))\n",
    "    train_en_sents_shuffled = train_en_sents[perm]\n",
    "    train_cn_sents_shuffled = train_cn_sents[perm]\n",
    "    train_cn_label_sents_shuffled = train_cn_label_sents[perm]\n",
    "\n",
    "    for iteration in range(train_en_sents_shuffled.shape[0] // batch_size):\n",
    "        x_data = train_en_sents_shuffled[(batch_size*iteration):(batch_size*(iteration+1))]\n",
    "        sent = paddle.to_tensor(x_data)\n",
    "        en_repr = encoder(sent)\n",
    "\n",
    "        x_cn_data = train_cn_sents_shuffled[(batch_size*iteration):(batch_size*(iteration+1))]\n",
    "        x_cn_label_data = train_cn_label_sents_shuffled[(batch_size*iteration):(batch_size*(iteration+1))]\n",
    "\n",
    "        # shape: (batch,  num_layer(=1 here) * num_of_direction(=1 here), hidden_size)\n",
    "        hidden = paddle.zeros([batch_size, 1, hidden_size])\n",
    "        cell = paddle.zeros([batch_size, 1, hidden_size])\n",
    "\n",
    "        loss = paddle.zeros([1])\n",
    "        # the decoder recurrent loop mentioned above\n",
    "        for i in range(MAX_LEN + 2):\n",
    "            cn_word = paddle.to_tensor(x_cn_data[:,i:i+1])\n",
    "            cn_word_label = paddle.to_tensor(x_cn_label_data[:,i:i+1])\n",
    "\n",
    "            logits, (hidden, cell) = atten_decoder(cn_word, hidden, cell, en_repr)\n",
    "            step_loss = F.softmax_with_cross_entropy(logits, cn_word_label)\n",
    "            avg_step_loss = paddle.mean(step_loss)\n",
    "            loss += avg_step_loss\n",
    "\n",
    "        loss = loss / (MAX_LEN + 2)\n",
    "        if(iteration % 200 == 0):\n",
    "            print(\"iter {}, loss:{}\".format(iteration, loss.numpy()))\n",
    "\n",
    "        loss.backward()\n",
    "        opt.step()\n",
    "        opt.clear_grad()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 使用模型进行机器翻译\n",
    "\n",
    "根据你所使用的计算设备的不同，上面的训练过程可能需要不等的时间。（在一台Mac笔记本上，大约耗时15~20分钟）\n",
    "完成上面的模型训练之后，我们可以得到一个能够从英文翻译成中文的机器翻译模型。接下来我们通过一个greedy search来实现使用该模型完成实际的机器翻译。（实际的任务中，你可能需要用beam search算法来提升效果）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "i want to study french\n",
      "true: 我要学法语。\n",
      "pred: 我要学法语。\n",
      "i didn t know that he was there\n",
      "true: 我不知道他在那裡。\n",
      "pred: 我不知道他在那裡。\n",
      "i called tom\n",
      "true: 我給湯姆打了電話。\n",
      "pred: 我看見湯姆了。\n",
      "he is getting along with his employees\n",
      "true: 他和他的員工相處。\n",
      "pred: 他和他的員工相處。\n",
      "we raced toward the fire\n",
      "true: 我們急忙跑向火。\n",
      "pred: 我們住在美國。\n",
      "i ran away in a hurry\n",
      "true: 我趕快跑走了。\n",
      "pred: 我在班里是最高。\n",
      "he cut the envelope open\n",
      "true: 他裁開了那個信封。\n",
      "pred: 他裁開了信封。\n",
      "he s shorter than tom\n",
      "true: 他比湯姆矮。\n",
      "pred: 他比湯姆矮。\n",
      "i ve just started playing tennis\n",
      "true: 我剛開始打網球。\n",
      "pred: 我剛去打網球。\n",
      "i need to go home\n",
      "true: 我该回家了。\n",
      "pred: 我该回家了。\n"
     ]
    }
   ],
   "source": [
    "encoder.eval()\n",
    "atten_decoder.eval()\n",
    "\n",
    "num_of_exampels_to_evaluate = 10\n",
    "\n",
    "indices = np.random.choice(len(train_en_sents),  num_of_exampels_to_evaluate, replace=False)\n",
    "x_data = train_en_sents[indices]\n",
    "sent = paddle.to_tensor(x_data)\n",
    "en_repr = encoder(sent)\n",
    "\n",
    "word = np.array(\n",
    "    [[cn_vocab['<bos>']]] * num_of_exampels_to_evaluate\n",
    ")\n",
    "word = paddle.to_tensor(word)\n",
    "\n",
    "hidden = paddle.zeros([num_of_exampels_to_evaluate, 1, hidden_size])\n",
    "cell = paddle.zeros([num_of_exampels_to_evaluate, 1, hidden_size])\n",
    "\n",
    "decoded_sent = []\n",
    "for i in range(MAX_LEN + 2):\n",
    "    logits, (hidden, cell) = atten_decoder(word, hidden, cell, en_repr)\n",
    "    word = paddle.argmax(logits, axis=1)\n",
    "    decoded_sent.append(word.numpy())\n",
    "    word = paddle.unsqueeze(word, axis=-1)\n",
    "    \n",
    "results = np.stack(decoded_sent, axis=1)\n",
    "for i in range(num_of_exampels_to_evaluate):\n",
    "    en_input = \" \".join(filtered_pairs[indices[i]][0])\n",
    "    ground_truth_translate = \"\".join(filtered_pairs[indices[i]][1])\n",
    "    model_translate = \"\"\n",
    "    for k in results[i]:\n",
    "        w = list(cn_vocab)[k]\n",
    "        if w != '<pad>' and w != '<eos>':\n",
    "            model_translate += w\n",
    "    print(en_input)\n",
    "    print(\"true: {}\".format(ground_truth_translate))\n",
    "    print(\"pred: {}\".format(model_translate))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The End\n",
    "\n",
    "你还可以通过变换网络结构，调整数据集，尝试不同的参数的方式来进一步提升本示例当中的机器翻译的效果。同时，也可以尝试在其他的类似的任务中用飞桨来完成实际的实践。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
