Merge branch 'develop' of upstream into infer_ckpt

fluid/DeepQNetwork/README.md

-# Reproduce DQN, DoubleDQN, DuelingDQN model with fluid version of PaddlePaddle
+[中文版](README_cn.md)
 
-+ DQN in:
+## Reproduce DQN, DoubleDQN, DuelingDQN model with Fluid version of PaddlePaddle
+Based on PaddlePaddle's next-generation API Fluid, the DQN model of deep reinforcement learning is reproduced, and the same level of indicators of the paper is reproduced in the classic Atari game. The model receives the image of the game as input, and uses the end-to-end model to directly predict the next step. The repository contains the following three types of models.
++ DQN in
 [Human-level Control Through Deep Reinforcement Learning](http://www.nature.com/nature/journal/v518/n7540/full/nature14236.html)
 + DoubleDQN in:
 [Deep Reinforcement Learning with Double Q-Learning](https://www.aaai.org/ocs/index.php/AAAI/AAAI16/paper/viewPaper/12389)
 + DuelingDQN in:
 [Dueling Network Architectures for Deep Reinforcement Learning](http://proceedings.mlr.press/v48/wangf16.html)
 
-# Atari benchmark & performance
-## [Atari games introduction](https://gym.openai.com/envs/#atari)
+## Atari benchmark & performance
+### [Atari games introduction](https://gym.openai.com/envs/#atari)
 
-+ Pong game result
+### Pong game result
+The average game rewards that can be obtained for the three models as the number of training steps changes during the training are as follows(about 3 hours/1 Million steps):
 ![DQN result](assets/dqn.png)
 
-# How to use
-+ Dependencies:
-    + python2.7
-    + gym
-    + tqdm
-    + paddlepaddle-gpu==0.12.0
-
-+ Start Training:
+## How to use
+### Dependencies:
++ python2.7
++ gym
++ tqdm
++ opencv-python
++ paddlepaddle-gpu>=0.12.0
++ ale_python_interface
+
+### Install Dependencies:
++ Install PaddlePaddle:
+    recommended to compile and install PaddlePaddle from source code
++ Install other dependencies:
+    ```
+    pip install -r requirement.txt
+    pip install gym[atari]
     ```
-    # To train a model for Pong game with gpu (use DQN model as default)
-    python train.py --rom ./rom_files/pong.bin --use_cuda
+    Install ale_python_interface, can reference：https://github.com/mgbellemare/Arcade-Learning-Environment
 
-    # To train a model for Pong with DoubleDQN
-    python train.py --rom ./rom_files/pong.bin --use_cuda --alg DoubleDQN
+### Start Training:
+```
+# To train a model for Pong game with gpu (use DQN model as default)
+python train.py --rom ./rom_files/pong.bin --use_cuda
 
-    # To train a model for Pong with DuelingDQN
-    python train.py --rom ./rom_files/pong.bin --use_cuda --alg DuelingDQN
-    ```
+# To train a model for Pong with DoubleDQN
+python train.py --rom ./rom_files/pong.bin --use_cuda --alg DoubleDQN
+
+# To train a model for Pong with DuelingDQN
+python train.py --rom ./rom_files/pong.bin --use_cuda --alg DuelingDQN
+```
 
 To train more games, can install more rom files from [here](https://github.com/openai/atari-py/tree/master/atari_py/atari_roms)
 
-+ Start Testing:
-    ```
-    # Play the game with saved model and calculate the average rewards
-    python play.py --rom ./rom_files/pong.bin --use_cuda --model_path ./saved_model/DQN-pong/stepXXXXX
+### Start Testing:
+```
+# Play the game with saved best model and calculate the average rewards
+python play.py --rom ./rom_files/pong.bin --use_cuda --model_path ./saved_model/DQN-pong
 
-    # Play the game with visualization
-    python play.py --rom ./rom_files/pong.bin --use_cuda --model_path ./saved_model/DQN-pong/stepXXXXX --viz 0.01
-    ```
+# Play the game with visualization
+python play.py --rom ./rom_files/pong.bin --use_cuda --model_path ./saved_model/DQN-pong --viz 0.01
+```
+[Here](https://pan.baidu.com/s/1gIsbNw5V7tMeb74ojx-TMA) is saved models for Pong and Breakout games. You can use it to play the game directly.

fluid/DeepQNetwork/README_cn.md

+## 基于PaddlePaddle的Fluid版本复现DQN, DoubleDQN, DuelingDQN三个模型
+基于PaddlePaddle下一代API Fluid复现了深度强化学习领域的DQN模型，在经典的Atari 游戏上复现了论文同等水平的指标，模型接收游戏的图像作为输入，采用端到端的模型直接预测下一步要执行的控制信号，本仓库一共包含以下3类模型。
++ DQN模型：
+[Human-level Control Through Deep Reinforcement Learning](http://www.nature.com/nature/journal/v518/n7540/full/nature14236.html)
++ DoubleDQN模型：
+[Deep Reinforcement Learning with Double Q-Learning](https://www.aaai.org/ocs/index.php/AAAI/AAAI16/paper/viewPaper/12389)
++ DuelingDQN模型：
+[Dueling Network Architectures for Deep Reinforcement Learning](http://proceedings.mlr.press/v48/wangf16.html)
+
+## 模型效果：Atari游戏表现
+### [Atari游戏介绍](https://gym.openai.com/envs/#atari)
+
+### Pong游戏训练结果
+三个模型在训练过程中随着训练步数的变化，能得到的平均游戏奖励如下图所示（大概3小时每1百万步）：
+![DQN result](assets/dqn.png)
+
+## 使用教程
+### 依赖:
++ python2.7
++ gym
++ tqdm
++ opencv-python
++ paddlepaddle-gpu>=0.12.0
++ ale_python_interface
+
+### 下载依赖：
++ 安装PaddlePaddle：
+    建议通过PaddlePaddle源码进行编译安装  
++ 下载其它依赖：
+    ```
+    pip install -r requirement.txt
+    pip install gym[atari]
+    ```
+    安装ale_python_interface可以参考：https://github.com/mgbellemare/Arcade-Learning-Environment
+
+### 训练模型：
+```
+# 使用GPU训练Pong游戏（默认使用DQN模型）
+python train.py --rom ./rom_files/pong.bin --use_cuda
+
+# 训练DoubleDQN模型
+python train.py --rom ./rom_files/pong.bin --use_cuda --alg DoubleDQN
+
+# 训练DuelingDQN模型
+python train.py --rom ./rom_files/pong.bin --use_cuda --alg DuelingDQN
+```
+
+训练更多游戏，可以下载游戏rom从[这里](https://github.com/openai/atari-py/tree/master/atari_py/atari_roms)
+
+### 测试模型：
+```
+# Play the game with saved model and calculate the average rewards
+# 使用训练过程中保存的最好模型玩游戏，以及计算平均奖励（rewards）
+python play.py --rom ./rom_files/pong.bin --use_cuda --model_path ./saved_model/DQN-pong
+
+# 以可视化的形式来玩游戏
+python play.py --rom ./rom_files/pong.bin --use_cuda --model_path ./saved_model/DQN-pong --viz 0.01
+```
+[这里](https://pan.baidu.com/s/1gIsbNw5V7tMeb74ojx-TMA)是Pong和Breakout游戏训练好的模型，可以直接用来测试。

fluid/DeepQNetwork/play.py

@@ -11,7 +11,7 @@ from tqdm import tqdm
 
 def predict_action(exe, state, predict_program, feed_names, fetch_targets,
                    action_dim):
-    if np.random.randint(100) == 0:
+    if np.random.random() < 0.01:
         act = np.random.randint(action_dim)
     else:
         state = np.expand_dims(state, axis=0)

fluid/DeepQNetwork/requirement.txt

+numpy
+gym
+tqdm
+opencv-python
+paddlepaddle-gpu==0.12.0

fluid/DeepQNetwork/train.py

@@ -120,6 +120,9 @@ def train_agent():
     pbar = tqdm(total=1e8)
     recent_100_reward = []
     total_step = 0
+    max_reward = None
+    save_path = os.path.join(args.model_dirname, '{}-{}'.format(
+        args.alg, os.path.basename(args.rom).split('.')[0]))
     while True:
         # start epoch
         total_reward, step = run_train_episode(agent, env, exp)
@@ -134,14 +137,11 @@ def train_agent():
             print("eval_agent done, (steps, eval_reward): ({}, {})".format(
                 total_step, eval_reward))
 
-        if total_step // args.save_every_steps == save_flag:
-            save_flag += 1
-            save_path = os.path.join(args.model_dirname, '{}-{}'.format(
-                args.alg, os.path.basename(args.rom).split('.')[0]),
-                                     'step{}'.format(total_step))
-            fluid.io.save_inference_model(save_path, ['state'],
-                                          agent.pred_value, agent.exe,
-                                          agent.predict_program)
+            if max_reward is None or eval_reward > max_reward:
+                max_reward = eval_reward
+                fluid.io.save_inference_model(save_path, ['state'],
+                                              agent.pred_value, agent.exe,
+                                              agent.predict_program)
     pbar.close()
 
 
@@ -173,11 +173,6 @@ if __name__ == '__main__':
         type=str,
         default='saved_model',
         help='dirname to save model')
-    parser.add_argument(
-        '--save_every_steps',
-        type=int,
-        default=100000,
-        help='every steps number to save model')
     parser.add_argument(
         '--test_every_steps',
         type=int,

fluid/README.cn.md

+# models 简介
+
+
+
+## 图像分类
+
+图像分类是根据图像的语义信息对不同类别图像进行区分，是计算机视觉中重要的基础问题，是物体检测、图像分割、物体跟踪、行为分析、人脸识别等其他高层视觉任务的基础，在许多领域都有着广泛的应用。如：安防领域的人脸识别和智能视频分析等，交通领域的交通场景识别，互联网领域基于内容的图像检索和相册自动归类，医学领域的图像识别等。
+
+在深度学习时代，图像分类的准确率大幅度提升，在图像分类任务中，我们向大家介绍了如何在经典的数据集ImageNet上，训练常用的模型，包括AlexNet、VGG、GoogLeNet、ResNet、Inception-v4、MobileNet、DPN(Dual Path Network)、SE-ResNeXt模型，也开源了[训练的模型](https://github.com/PaddlePaddle/models/blob/develop/fluid/image_classification/README_cn.md#已有模型及其性能)方便用户下载使用。同时提供了能够将Caffe模型转换为PaddlePaddle Fluid模型配置和参数文件的工具。
+
+- [AlexNet](https://github.com/PaddlePaddle/models/tree/develop/fluid/image_classification/models)
+- [VGG](https://github.com/PaddlePaddle/models/tree/develop/fluid/image_classification/models)
+- [GoogleNet](https://github.com/PaddlePaddle/models/tree/develop/fluid/image_classification/models)
+- [Residual Network](https://github.com/PaddlePaddle/models/tree/develop/fluid/image_classification/models)
+- [Inception-v4](https://github.com/PaddlePaddle/models/tree/develop/fluid/image_classification/models)
+- [MobileNet](https://github.com/PaddlePaddle/models/tree/develop/fluid/image_classification/models)
+- [Dual Path Network](https://github.com/PaddlePaddle/models/tree/develop/fluid/image_classification/models)
+- [SE-ResNeXt](https://github.com/PaddlePaddle/models/tree/develop/fluid/image_classification/models)
+- [Caffe模型转换为Paddle Fluid配置和模型文件工具](https://github.com/PaddlePaddle/models/tree/develop/fluid/image_classification/caffe2fluid)
+
+## 目标检测
+
+目标检测任务的目标是给定一张图像或是一个视频帧，让计算机找出其中所有目标的位置，并给出每个目标的具体类别。对于人类来说，目标检测是一个非常简单的任务。然而，计算机能够“看到”的是图像被编码之后的数字，很难解图像或是视频帧中出现了人或是物体这样的高层语义概念，也就更加难以定位目标出现在图像中哪个区域。与此同时，由于目标会出现在图像或是视频帧中的任何位置，目标的形态千变万化，图像或是视频帧的背景千差万别，诸多因素都使得目标检测对计算机来说是一个具有挑战性的问题。
+
+在目标检测任务中，我们介绍了如何基于[PASCAL VOC](http://host.robots.ox.ac.uk/pascal/VOC/)、[MS COCO](http://cocodataset.org/#home)数据的训练目标检测算法SSD，SSD全称Single Shot MultiBox Detector，是目标检测领域较新且效果较好的检测算法之一，具有检测速度快且检测精度高的特点，并开源了训练好的[MobileNet-SSD模型](https://github.com/PaddlePaddle/models/blob/develop/fluid/object_detection/README_cn.md#模型发布)。
+
+- [Single Shot MultiBox Detector](https://github.com/PaddlePaddle/models/blob/develop/fluid/object_detection/README_cn.md)
+
+
+## 图像语义分割
+
+图像语意分割顾名思义是将图像像素按照表达的语义含义的不同进行分组/分割，图像语义是指对图像内容的理解，例如，能够描绘出什么物体在哪里做了什么事情等，分割是指对图片中的每个像素点进行标注，标注属于哪一类别。近年来用在无人车驾驶技术中分割街景来避让行人和车辆、医疗影像分析中辅助诊断等。
+
+在图像语义分割任务中，我们介绍如何基于图像级联网络(Image Cascade Network,ICNet)进行语义分割，相比其他分割算法，ICNet兼顾了准确率和速度。
+
+
+- [ICNet](https://github.com/PaddlePaddle/models/tree/develop/fluid/icnet)
+
+
+## 场景文字识别
+
+许多场景图像中包含着丰富的文本信息，对理解图像信息有着重要作用，能够极大地帮助人们认知和理解场景图像的内容。场景文字识别是在图像背景复杂、分辨率低下、字体多样、分布随意等情况下，将图像信息转化为文字序列的过程，可认为是一种特别的翻译过程：将图像输入翻译为自然语言输出。场景图像文字识别技术的发展也促进了一些新型应用的产生，如通过自动识别路牌中的文字帮助街景应用获取更加准确的地址信息等。
+
+在场景文字识别任务中，我们介绍如何将基于CNN的图像特征提取和基于RNN的序列翻译技术结合，免除人工定义特征，避免字符分割，使用自动学习到的图像特征，完成端到端地无约束字符定位和识别。当前，介绍了CRNN-CTC模型，后续会引入基于注意力机制的序列到序列模型。
+
+- [CRNN-CTC模型](https://github.com/PaddlePaddle/models/tree/develop/fluid/ocr_recognition)