Merge pull request #94 from hysunflower/update_mpc_scripts

Update mpc scripts

python/paddle_fl/mpc/examples/mnist_demo/README.md

@@ -20,6 +20,15 @@ Encrypted data files of feature and label would be generated and saved in `/tmp`
 
 #### (2). Launch Demo with A Shell Script
 
+You should set the env params as follow:
+
+```
+export PYTHON=/yor/python
+export PATH_TO_REDIS_BIN=/path/to/redis_bin
+export LOCALHOST=/your/localhost
+export REDIS_PORT=/your/redis/port
+```
+
 Launch demo with the `run_standalone.sh` script. The concrete command is:
 
 ```bash
@@ -32,11 +41,14 @@ Besides, predictions would be made in this demo once training is finished. The p
 
 #### (3). Decrypt Data
 
-Decrypt the saved prediction data and save the decrypted prediction results into a specified file using `decrypt_data_to_file()` in `process_data.py` script. For example, users can write the following code into a python script named `decrypt_save.py`, and then run the script with command `python decrypt_save.py`. The decrypted prediction results would be saved into `mpc_label`.
+Decrypt the saved prediction data and save the decrypted prediction results into a specified file using `decrypt_data_to_file()` in `process_data.py` script. For example, users can write the following code into a python script named `decrypt_save.py`, and then run the script with command `python decrypt_save.py decrypt_file`. The decrypted prediction results would be saved into `decrypt_file`.
 
 ```python
+import sys
+
+decrypt_file=sys.argv[1]
 import process_data
-process_data.decrypt_data_to_file("/tmp/mnist_output_prediction", (BATCH_SIZE,), "mpc_label")
+process_data.decrypt_data_to_file("/tmp/mnist_output_prediction", (BATCH_SIZE,), decrypt_file)
 ```
 
 **Note** that remember to delete the prediction files in `/tmp` directory generated in last running, in case of any influence on the decrypted results of current running. For simplifying users operations, we provide the following commands in `run_standalone.sh`, which can delete the files mentioned above when running this script.
@@ -91,10 +103,13 @@ Similarly, predictions with cypher text format would be saved in `/tmp` director
 
 #### (5). Decrypt Prediction Data
 
-Each computation party sends  `mnist_output_prediction.part` file in `/tmp` directory to the `/tmp` directory of data owner. Data owner decrypts the prediction data and saves the decrypted prediction results into a specified file using `decrypt_data_to_file()` in `process_data.py` script. For example, users can write the following code into a python script named `decrypt_save.py`, and then run the script with command `python decrypt_save.py`. The decrypted prediction results would be saved into file `mpc_label`.
+Each computation party sends  `mnist_output_prediction.part` file in `/tmp` directory to the `/tmp` directory of data owner. Data owner decrypts the prediction data and saves the decrypted prediction results into a specified file using `decrypt_data_to_file()` in `process_data.py` script. For example, users can write the following code into a python script named `decrypt_save.py`, and then run the script with command `python decrypt_save.py decrypt_file`. The decrypted prediction results would be saved into file `decrypt_file`.
 
 ```python
+import sys
+
+decrypt_file=sys.argv[1]
 import process_data
-process_data.decrypt_data_to_file("/tmp/mnist_output_prediction", (BATCH_SIZE,), "mpc_label")
+process_data.decrypt_data_to_file("/tmp/mnist_output_prediction", (BATCH_SIZE,), decrypt_file)
 ```
 

python/paddle_fl/mpc/examples/mnist_demo/README_CN.md

@@ -20,7 +20,16 @@ process_data.generate_encrypted_test_data()
 
 #### 2. 使用shell脚本启动demo
 
-使用`run_standalone.sh`脚本，启动并运行demo，命令如下：
+运行demo之前，需设置以下环境变量：
+
+```
+export PYTHON=/yor/python
+export PATH_TO_REDIS_BIN=/path/to/redis_bin
+export LOCALHOST=/your/localhost
+export REDIS_PORT=/your/redis/port
+```
+
+然后使用`run_standalone.sh`脚本，启动并运行demo，命令如下：
 
 ```bash 
 bash run_standalone.sh mnist_demo.py
@@ -32,11 +41,14 @@ bash run_standalone.sh mnist_demo.py
 
 #### 3. 解密数据
 
-使用`process_data.py`脚本中的`decrypt_data_to_file()`，将保存的密文预测结果进行解密，并且将解密得到的明文预测结果保存到指定文件中。例如，将下面的内容写到一个`decrypt_save.py`脚本中，然后`python decrypt_save.py`，将把明文预测结果保存在`mpc_label`文件中。
+使用`process_data.py`脚本中的`decrypt_data_to_file()`，将保存的密文预测结果进行解密，并且将解密得到的明文预测结果保存到指定文件中。例如，将下面的内容写到一个`decrypt_save.py`脚本中，然后`python decrypt_save.py decrypt_file`，将把明文预测结果保存在`decrypt_file`文件中。
 
 ```python
+import sys
+
+decrypt_file=sys.argv[1]
 import process_data
-process_data.decrypt_data_to_file("/tmp/mnist_output_prediction", (BATCH_SIZE,), "mpc_label")
+process_data.decrypt_data_to_file("/tmp/mnist_output_prediction", (BATCH_SIZE,), decrypt_file)
 ```
 
 **注意**：再次启动运行demo之前，请先将上次在`/tmp`保存的预测密文结果文件删除，以免影响本次密文数据的恢复结果。为了简化用户操作，我们在`run_standalone.sh`脚本中加入了如下的内容，可以在执行脚本时删除上次的数据。
@@ -93,10 +105,13 @@ $PYTHON_EXECUTABLE mnist_demo.py $PARTY_ID $SERVER $PORT
 
 #### 5. 解密预测数据
 
-各计算party将`/tmp`目录下的`mnist_output_prediction.part`文件发送到数据方的/tmp目录下。数据方使用`process_data.py`脚本中的`decrypt_data_to_file()`，将密文预测结果进行解密，并且将解密得到的明文预测结果保存到指定文件中。例如，将下面的内容写到一个`decrypt_save.py`脚本中，然后`python decrypt_save.py`，将把明文预测结果保存在`mpc_label`文件中。
+各计算party将`/tmp`目录下的`mnist_output_prediction.part`文件发送到数据方的/tmp目录下。数据方使用`process_data.py`脚本中的`decrypt_data_to_file()`，将密文预测结果进行解密，并且将解密得到的明文预测结果保存到指定文件中。例如，将下面的内容写到一个`decrypt_save.py`脚本中，然后`python decrypt_save.py decrypt_file`，将把明文预测结果保存在`decrypt_file`文件中。
 
 ```python
+import sys
+
+decrypt_file=sys.argv[1]
 import process_data
-process_data.decrypt_data_to_file("/tmp/mnist_output_prediction", (BATCH_SIZE,), "mpc_label")
+process_data.decrypt_data_to_file("/tmp/mnist_output_prediction", (BATCH_SIZE,), decrypt_file)
 ```
 

python/paddle_fl/mpc/examples/mnist_demo/decrypt_save.py

+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""
+Decrypt Prediction Data.
+"""
+import sys
+
+import process_data
+
+decrypt_file=sys.argv[1]
+BATCH_SIZE=128
+process_data.decrypt_data_to_file("/tmp/mnist_output_prediction", (BATCH_SIZE,), decrypt_file)

python/paddle_fl/mpc/examples/mnist_demo/mnist_demo.py

@@ -16,6 +16,7 @@ MNIST Demo
 """
 
 import sys
+import os
 
 import numpy as np
 import time
@@ -78,18 +79,20 @@ test_loader.set_batch_generator(test_batch_sample, places=place)
 exe = fluid.Executor(place)
 exe.run(fluid.default_startup_program())
 
-start_time = time.time()
-step = 0
 for epoch_id in range(epoch_num):
+    start_time = time.time()
+    step = 0
     # feed data via loader
     for sample in loader():
+        batch_start = time.time()
         exe.run(feed=sample, fetch_list=[cost.name])
+        batch_end = time.time()
         if step % 50 == 0:
-            print('Epoch={}, Step={}'.format(epoch_id, step))
+            print('Epoch={}, Step={}, batch_cost={:.4f} s'.format(epoch_id, step, (batch_end - batch_start)))
         step += 1
 
-end_time = time.time()
-print('Mpc Training of Epoch={} Batch_size={}, cost time in seconds:{}'
+    end_time = time.time()
+    print('Mpc Training of Epoch={} Batch_size={}, epoch_cost={:.4f} s'
       .format(epoch_num, BATCH_SIZE, (end_time - start_time)))
 
 # prediction

python/paddle_fl/mpc/examples/mnist_demo/prepare.py

+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""
+Prepare data for MNIST.
+"""
+import process_data
+
+
+process_data.generate_encrypted_data()
+process_data.generate_encrypted_test_data()

python/paddle_fl/mpc/examples/mnist_demo/run_standalone.sh

@@ -31,12 +31,13 @@
 # bash run_standalone.sh TEST_SCRIPT_NAME
 #
 
-# modify the following vars according to your environment
-PYTHON="python"
-REDIS_HOME="path_to_redis_bin"
-SERVER="localhost"
-PORT=9937
+# please set the following environment vars according in your environment
+PYTHON=${PYTHON}
+REDIS_HOME=${PATH_TO_REDIS_BIN}
+SERVER=${LOCALHOST}
+PORT=${REDIS_PORT}
 
+echo "redis home in ${REDIS_HOME}, server is ${SERVER}, port is ${PORT}"
 function usage() {
     echo 'run_standalone.sh SCRIPT_NAME [ARG...]'
     exit 0
@@ -63,10 +64,21 @@ $REDIS_BIN -h $SERVER -p $PORT flushall
 
 # remove temp data generated in last time
 PRED_FILE="/tmp/mnist_output_prediction.*"
-if [ "$PRED_FILE" ]; then
+ls ${PRED_FILE}
+if [ $? -eq 0 ]; then
         rm -rf $PRED_FILE
 fi
 
+TRAINING_FILE="/tmp/mnist2_feature.part*"
+ls ${TRAINING_FILE}
+if [ $? -ne 0 ]; then
+    echo "There is no data in /tmp, please prepare data with "python prepare.py" firstly"
+    exit 1
+else
+    echo "There are data for mnist:"
+    echo "`ls ${TRAINING_FILE}`"
+fi
+
 
 # kick off script with roles of 1 and 2, and redirect output to /dev/null
 for role in {1..2}; do