Skip to content

P
PaddleFL

PaddlePaddle / PaddleFL

通知 35
Star 5
Fork 1
P
PaddleFL
提交 c069531d 编写于 作者: J jingqinghe
浏览文件
操作

update README and document

上级 7d144c3b
隐藏空白更改
内联 并排
Showing with 558 addition and 315 deletion
README.md
浏览文件 @ c069531d
......@@ -11,15 +11,13 @@ Data is becoming more and more expensive nowadays, and sharing of raw data is ve
## Overview of PaddleFL
### Horizontal Federated Learning
<img src='images/FL-framework.png' width = "1000" height = "320" align="middle"/>
In PaddleFL, horizontal and vertical federated learning strategies will be implemented according to the categorization given in [4]. Application demonstrations in natural language processing, computer vision and recommendation will be provided in PaddleFL.
#### A. Federated Learning Strategy
- **Vertical Federated Learning**: Logistic Regression with PrivC[5], Neural Network with ABY3 [11]
- **Vertical Federated Learning**: Logistic Regression with PrivC[5], Neural Network with MPC [11]
- **Horizontal Federated Learning**: Federated Averaging [2], Differential Privacy [6], Secure Aggregation
......@@ -31,19 +29,15 @@ In PaddleFL, horizontal and vertical federated learning strategies will be imple
- **Active Learning**
### Federated Learning with MPC
<img src='images/PFM-overview.png' width = "1000" height = "446" align="middle"/>
There are mainly two components in PaddleFL: **Data Parallel** and **Federated Learning with MPC (PFM)**.
Paddle FL MPC (PFM) is a framework for privacy-preserving deep learning based on PaddlePaddle. It follows the same running mechanism and programming paradigm with PaddlePaddle, while using secure multi-party computation (MPC) to enable secure training and prediction.
With Data Parallel, distributed data holders can finish their Federated Learning tasks based on common horizontal federated strategies, such as FedAvg, DPSGD, etc.
With PFM, it is easy to train models or conduct prediction as on PaddlePaddle over encrypted data, without the need for cryptography expertise. Furthermore, the rich industry-oriented models and algorithms built on PaddlePaddle can be smoothly migrated to secure versions on PFM with little effort.
Besides, PFM is implemented based on secure multi-party computation (MPC) to enable secure training and prediction. As a key product of PaddleFL, PFM intrinsically supports federated learning well, including horizontal, vertical and transfer learning scenarios. Users with little cryptography expertise can also train models or conduct prediction on encrypted data.
As a key product of PaddleFL, PFM intrinsically supports federated learning well, including horizontal, vertical and transfer learning scenarios. It provides both provable security (semantic security) and competitive performance.
## Installation
## Compilation and Installation
### Docker Installation
We **highly recommend** to run PaddleFL in Docker
```sh
#Pull and run the docker
......@@ -54,61 +48,39 @@ docker run --name <docker_name> --net=host -it -v $PWD:/root <image id> /bin/bas
pip install paddle_fl
```
### Compile From Source Code
#### A. Environment preparation
* CentOS 6 or CentOS 7 (64 bit)
* Python 2.7.15+/3.5.1+/3.6/3.7 ( 64 bit) or above
* pip or pip3 9.0.1+ (64 bit)
* PaddlePaddle release 1.8
* Redis 5.0.8 (64 bit)
* GCC or G++ 4.8.3+
* cmake 3.15+
If you want to compile and install from source code, please click [here](./docs/source/md/compile_and_install.md) to get instructions.
#### B. Clone the source code, compile and install
We also prepare a stable redis package for you to download and install, which will be used in tasks with MPC.
Fetch the source code and checkout stable release
```sh
git clone https://github.com/PaddlePaddle/PaddleFL
cd /path/to/PaddleFL
# Checkout stable release
mkdir build && cd build
wget --no-check-certificate https://paddlefl.bj.bcebos.com/redis-stable.tar
tar -xf redis-stable.tar
cd redis-stable && make
```
Execute compile commands, where `PYTHON_EXECUTABLE` is path to the python binary where the PaddlePaddle is installed, `CMAKE_CXX_COMPILER` is the path of G++ and `PYTHON_INCLUDE_DIRS` is the corresponding python include directory. You can get the `PYTHON_INCLUDE_DIRS` via the following command:
## Easy deployment with kubernetes
### Data Parallel
```sh
${PYTHON_EXECUTABLE} -c "from distutils.sysconfig import get_python_inc;print(get_python_inc())"
```
Then you can put the directory in the following command and make:
```sh
cmake ../ -DPYTHON_EXECUTABLE=${PYTHON_EXECUTABLE} -DPYTHON_INCLUDE_DIRS=${python_include_dir} -DCMAKE_CXX_COMPILER=${g++_path}
make -j$(nproc)
```
Install the package:
```sh
make install
cd /path/to/PaddleFL/python
${PYTHON_EXECUTABLE} setup.py sdist bdist_wheel
pip or pip3 install dist/***.whl -U
```
We also prepare a stable redis package for you to download and install
kubectl apply -f ./python/paddle_fl/paddle_fl/examples/k8s_deployment/master.yaml
```sh
wget --no-check-certificate https://paddlefl.bj.bcebos.com/redis-stable.tar
tar -xf redis-stable.tar
cd redis-stable && make
```
Please refer [K8S deployment example](./python/paddle_fl/paddle_fl/examples/k8s_deployment/README.md) for details
You can also refer [K8S cluster application and kubectl installation](./python/paddle_fl/paddle_fl/examples/k8s_deployment/deploy_instruction.md) to deploy your K8S cluster
### Federated Learning with MPC
To be added.
## Framework design of PaddleFL
### Horizontal Federated Learning
### Data Parallel
<img src='images/FL-training.png' width = "1000" height = "400" align="middle"/>
In PaddleFL, components for defining a federated learning task and training a federated learning job are as follows:
In Data Parallel, components for defining a federated learning task and training a federated learning job are as follows:
#### A. Compile Time
......@@ -126,106 +98,47 @@ In PaddleFL, components for defining a federated learning task and training a fe
- **FL-Worker**: Each organization participates in federated learning will have one or more federated workers that will communicate with the federated parameter server.
- **FL-scheduler**: Decide which set of trainers can join the training before each updating cycle.
- **FL-scheduler**: Decide which set of trainers can join the training before each updating cycle.
For more instructions, please refer to the [examples](./python/paddle_fl/paddle_fl/examples)
### Federated Learning with MPC
<img src='images/PFM-overview.png' width = "1000" height = "446" align="middle"/>
Paddle FL MPC implements secure training and inference tasks based on the underlying MPC protocol like ABY3[11], which is a high efficient three-party computing model.
In ABY3, participants can be classified into roles of Input Party (IP), Computing Party (CP) and Result Party (RP). Input Parties (e.g., the training data/model owners) encrypt and distribute data or models to Computing Parties. Computing Parties (e.g., the VM on the cloud) conduct training or inference tasks based on specific MPC protocols, being restricted to see only the encrypted data or models, and thus guarantee the data privacy. When the computation is completed, one or more Result Parties (e.g., data owners or specified third-party) receive the encrypted results from Computing Parties, and reconstruct the plaintext results. Roles can be overlapped, e.g., a data owner can also act as a computing party.
In ABY3, participants can be classified into roles of Input Party (IP), Computing Party (CP) and Result Party (RP). Input Parties (e.g., the training data/model owners) encrypt and distribute data or models to Computing Parties. Computing Parties (e.g., the VM on the cloud) conduct training or inference tasks based on specific MPC protocols, being restricted to see only the encrypted data or models, and thus guarantee the data privacy. When the computation is completed, one or more Result Parties (e.g., data owners or specified third-party) receive the encrypted results from Computing Parties, and reconstruct the plaintext results. Roles can be overlapped, e.g., a data owner can also act as a computing party.
A full training or inference process in PFM consists of mainly three phases: data preparation, training/inference, and result reconstruction.
#### A. Data preparation
##### 1. Private data alignment
PFM enables data owners (IPs) to find out records with identical keys (like UUID) without revealing private data to each other. This is especially useful in the vertical learning cases where segmented features with same keys need to be identified and aligned from all owners in a private manner before training. Using the OT-based PSI (Private Set Intersection) algorithm, PFM can perform private alignment at a speed of up to 60k records per second.
##### 2. Encryption and distribution
- **Private data alignment**: PFM enables data owners (IPs) to find out records with identical keys (like UUID) without revealing private data to each other. This is especially useful in the vertical learning cases where segmented features with same keys need to be identified and aligned from all owners in a private manner before training.
In PFM, data and models from IPs will be encrypted using Secret-Sharing[10], and then be sent to CPs, via directly transmission or distributed storage like HDFS. Each CP can only obtain one share of each piece of data, and thus is unable to recover the original value in the Semi-honest model.
- **Encryption and distribution**: In PFM, data and models from IPs will be encrypted using Secret-Sharing[10], and then be sent to CPs, via directly transmission or distributed storage like HDFS. Each CP can only obtain one share of each piece of data, and thus is unable to recover the original value in the Semi-honest model.
#### B. Training/inference
<img src='images/PFM-design.png' width = "1000" height = "622" align="middle"/>
A PFM program is exactly a PaddlePaddle program, and will be executed as normal PaddlePaddle programs. Before training/inference, user needs to choose a MPC protocol, define a machine learning model and their training strategies. Typical machine learning operators are provided in `paddle_fl.mpc` over encrypted data, of which the instances are created and run in order by Executor during run-time.
As in PaddlePaddle, a training or inference job can be separated into the compile-time phase and the run-time phase:
##### 1. Compile time
* **MPC environment specification**: a user needs to choose a MPC protocol, and configure the network settings. In current version, PFM provides only the "ABY3" protocol. More protocol implementation will be provided in future.
* **User-defined job program**: a user can define the machine learning model structure and the training strategies (or inference task) in a PFM program, using the secure operators.
##### 2. Run time
A PFM program is exactly a PaddlePaddle program, and will be executed as normal PaddlePaddle programs. For example, in run-time a PFM program will be transpiled into ProgramDesc, and then be passed to and run by the Executor. The main concepts in the run-time phase are as follows:
* **Computing nodes**: a computing node is an entity corresponding to a Computing Party. In real deployment, it can be a bare-metal machine, a cloud VM, a docker or even a process. PFM requires exactly three computing nodes in each run, which is determined by the underlying ABY3 protocol. A PFM program will be deployed and run in parallel on all three computing nodes.
* **Operators using MPC**: PFM provides typical machine learning operators in `paddle_fl.mpc` over encrypted data. Such operators are implemented upon PaddlePaddle framework, based on MPC protocols like ABY3. Like other PaddlePaddle operators, in run time, instances of PFM operators are created and run in order by Executor.
For more information of Training/inference phase, please refer to the following [doc](./docs/source/md/mpc_train.md).
#### C. Result reconstruction
Upon completion of the secure training (or inference) job, the models (or prediction results) will be output by CPs in encrypted form. Result Parties can collect the encrypted results, decrypt them using the tools in PFM, and deliver the plaintext results to users.
For more instructions, please refer to [mpc examples](./python/paddle_fl/mpc/examples)
## Easy deployment with kubernetes
### Horizontal Federated Learning
```sh
kubectl apply -f ./python/paddle_fl/paddle_fl/examples/k8s_deployment/master.yaml
```
Please refer [K8S deployment example](./python/paddle_fl/paddle_fl/examples/k8s_deployment/README.md) for details
You can also refer [K8S cluster application and kubectl installation](./python/paddle_fl/paddle_fl/examples/k8s_deployment/deploy_instruction.md) to deploy your K8S cluster
### Federated Learning with MPC
To be added.
For more instructions, please refer to [mpc examples](./python/paddle_fl/mpc/examples)
## Benchmark task
### Horzontal Federated Learning
### Data Parallel
Gru4Rec [9] introduces recurrent neural network model in session-based recommendation. PaddlePaddle's Gru4Rec implementation is in https://github.com/PaddlePaddle/models/tree/develop/PaddleRec/gru4rec. An example is given in [Gru4Rec in Federated Learning](https://paddlefl.readthedocs.io/en/latest/examples/gru4rec_examples.html)
### Federated Learning with MPC
#### A. Convergence of paddle_fl.mpc vs paddle
##### 1. Training Parameters
- Dataset: Boston house price dataset
- Number of Epoch: 20
- Batch Size: 10
##### 2. Experiment Results
| Epoch/Step | paddle_fl.mpc | Paddle |
| ---------- | ------------- | ------ |
| Epoch=0, Step=0 | 738.39491 | 738.46204 |
| Epoch=1, Step=0 | 630.68834 | 629.9071 |
| Epoch=2, Step=0 | 539.54683 | 538.1757 |
| Epoch=3, Step=0 | 462.41159 | 460.64722 |
| Epoch=4, Step=0 | 397.11516 | 395.11017 |
| Epoch=5, Step=0 | 341.83102 | 339.69815 |
| Epoch=6, Step=0 | 295.01114 | 292.83597 |
| Epoch=7, Step=0 | 255.35141 | 253.19429 |
| Epoch=8, Step=0 | 221.74739 | 219.65132 |
| Epoch=9, Step=0 | 193.26459 | 191.25981 |
| Epoch=10, Step=0 | 169.11423 | 167.2204 |
| Epoch=11, Step=0 | 148.63138 | 146.85835 |
| Epoch=12, Step=0 | 131.25081 | 129.60391 |
| Epoch=13, Step=0 | 116.49708 | 114.97599 |
| Epoch=14, Step=0 | 103.96669 | 102.56854 |
| Epoch=15, Step=0 | 93.31706 | 92.03858 |
| Epoch=16, Step=0 | 84.26219 | 83.09653 |
| Epoch=17, Step=0 | 76.55664 | 75.49785 |
| Epoch=18, Step=0 | 69.99673 | 69.03561 |
| Epoch=19, Step=0 | 64.40562 | 63.53539 |
We conduct tests on PFM using Boston house price dataset, and the implementation is given in [uci_demo](./python/paddle_fl/mpc/examples/uci_demo)
## On Going and Future Work
......@@ -233,6 +146,8 @@ Gru4Rec [9] introduces recurrent neural network model in session-based recommend
- Add K8S deployment scheme for Paddle Encrypted.
- FL mobile simulator will be open sourced in following versions.
## Reference
[1]. Jakub Konečný, H. Brendan McMahan, Daniel Ramage, Peter Richtárik. **Federated Optimization: Distributed Machine Learning for On-Device Intelligence.** 2016
......
README_cn.md
浏览文件 @ c069531d
# PaddleFL
PaddleFL是一个基于PaddlePaddle的开源联邦学习框架。研究人员可以很轻松地用PaddleFL复制和比较不同的联邦学习算法。开发人员也可以从padderFL中获益，因为用PaddleFL在大规模分布式集群中部署联邦学习系统很容易。PaddleFL提供很多联邦学习策略及其在计算机视觉、自然语言处理、推荐算法等领域的应用。此外，PaddleFL还将提供传统机器学习训练策略的应用，例如多任务学习、联邦学习环境下的迁移学习。依靠着PaddlePaddle的大规模分布式训练和Kubernetes对训练任务的弹性调度能力，PaddleFL可以基于全栈开源软件轻松地部署。
PaddleFL是一个基于PaddlePaddle的开源联邦学习框架。研究人员可以很轻松地用PaddleFL复制和比较不同的联邦学习算法。开发人员也可以从paddleFL中获益，因为用PaddleFL在大规模分布式集群中部署联邦学习系统很容易。PaddleFL提供很多联邦学习策略及其在计算机视觉、自然语言处理、推荐算法等领域的应用。此外，PaddleFL还将提供传统机器学习训练策略的应用，例如多任务学习、联邦学习环境下的迁移学习。依靠着PaddlePaddle的大规模分布式训练和Kubernetes对训练任务的弹性调度能力，PaddleFL可以基于全栈开源软件轻松地部署。
## 联邦学习
......@@ -8,8 +8,6 @@ PaddleFL是一个基于PaddlePaddle的开源联邦学习框架。研究人员可
## PaddleFL概述
### 横向联邦方案
<img src='images/FL-framework-zh.png' width = "1300" height = "310" align="middle"/>
在PaddleFL中，横向和纵向联邦学习策略将根据[4]中给出的分类来实现。PaddleFL也将提供在自然语言处理，计算机视觉和推荐算法等领域的应用示例。
......@@ -28,19 +26,15 @@ PaddleFL是一个基于PaddlePaddle的开源联邦学习框架。研究人员可
- **主动学习**
### Federated Learning with MPC
PaddleFL 中主要提供两种解决方案：**Data Parallel** 以及 **Federated Learning with MPC (PFM)**。
<img src='images/PFM-overview.png' width = "1000" height = "446" align="middle"/>
通过Data Parallel，各数据方可以基于经典的横向联邦学习策略（如 FedAvg，DPSGD等）完成模型训练。
Paddle FL MPC(PFM) 是一个基于PaddlePaddle的隐私保护深度学习框架。Paddle Encrypted基于多方计算（MPC）实现安全训练及预测，拥有与PaddlePaddle相同的运行机制及编程范式。
此外，PFM是基于多方安全计算（MPC）实现的联邦学习方案。作为PaddleFL的一个重要组成部分，PFM可以很好地支持联邦学习，包括横向、纵向及联邦迁移学习等多个场景。既提供了可靠的安全性，也拥有可观的性能。
PFM 设计与PaddlePaddle相似，没有密码学相关背景的用户亦可简单的对加密的数据进行训练和预测。同时，PaddlePaddle中丰富的模型和算法可以轻易地迁移到PFM中。
## 安装
作为PaddleFL的一个重要组成部分，PFM可以很好地支持联邦学习，包括横向、纵向及联邦迁移学习等多个场景。既提供了可靠的安全性，也拥有可观的性能。
## 编译与安装
### 使用docker安装
我们**强烈建议** 您在docker中使用PaddleFL。
```sh
#Pull and run the docker
......@@ -51,59 +45,35 @@ docker run --name <docker_name> --net=host -it -v $PWD:/root <image id> /bin/bas
pip install paddle_fl
```
### 从源码编译
#### A. 环境准备
若您希望从源码编译安装，请点击[这里](./docs/source/md/compile_and_install_cn.md)。
* CentOS 6 or CentOS 7 (64 bit)
* Python 2.7.15+/3.5.1+/3.6/3.7 ( 64 bit) or above
* pip or pip3 9.0.1+ (64 bit)
* PaddlePaddle release 1.8
* Redis 5.0.8 (64 bit)
* GCC or G++ 4.8.3+
* cmake 3.15+
#### B. 克隆源代码并编译安装
我们也提供了稳定的redis安装包, 可供下载。
获取源代码
```sh
git clone https://github.com/PaddlePaddle/PaddleFL
cd /path/to/PaddleFL
# Checkout stable release
mkdir build && cd build
wget --no-check-certificate https://paddlefl.bj.bcebos.com/redis-stable.tar
tar -xf redis-stable.tar
cd redis-stable && make
```
执行编译指令, `PYTHON_EXECUTABLE` 为安装了PaddlePaddle的可执行python路径, `CMAKE_CXX_COMPILER` 为指定的g++路径。 `PYTHON_INCLUDE_DIRS` 是相应的include路径，可以用如下指令获得：
## Kubernetes简单部署
### 横向联邦方案
```sh
${PYTHON_EXECUTABLE} -c "from distutils.sysconfig import get_python_inc;print(get_python_inc())"
```
之后就可以执行编译和安装的指令
```sh
cmake ../ -DPYTHON_EXECUTABLE=${PYTHON_EXECUTABLE} -DPYTHON_INCLUDE_DIRS=${python_include_dir} -DCMAKE_CXX_COMPILER=${g++_path}
make -j$(nproc)
```
安装对应的安装包
```sh
make install
cd /path/to/PaddleFL/python
${PYTHON_EXECUTABLE} setup.py sdist bdist_wheel
pip or pip3 install dist/***.whl -U
kubectl apply -f ./python/paddle_fl/paddle_fl/examples/k8s_deployment/master.yaml
```
请参考[K8S部署实例](./python/paddle_fl/paddle_fl/examples/k8s_deployment/README.md)
我们也提供了稳定的redis安装包, 可供下载。
也可以参考[K8S集群申请及kubectl安装](./python/paddle_fl/paddle_fl/examples/k8s_deployment/deploy_instruction.md) 配置自己的K8S集群
```sh
wget --no-check-certificate https://paddlefl.bj.bcebos.com/redis-stable.tar
tar -xf redis-stable.tar
cd redis-stable && make
```
### Federated Learning with MPC
会在后续版本中发布。
## PaddleFL框架设计
### 横向联邦方案
### Data Parallel
<img src='images/FL-training.png' width = "1300" height = "450" align="middle"/>
......@@ -131,101 +101,44 @@ cd redis-stable && make
### Federated Learning with MPC
<img src='images/PFM-overview.png' width = "1000" height = "446" align="middle"/>
Paddle FL MPC 中的安全训练和推理任务是基于高效的多方计算协议实现的，如ABY3[11]
在ABY3[11]中，参与方可分为：输入方、计算方和结果方。输入方为训练数据及模型的持有方，负责加密数据和模型，并将其发送到计算方。计算方为训练的执行方，基于特定的多方安全计算协议完成训练任务。计算方只能得到加密后的数据及模型，以保证数据隐私。计算结束后，结果方会拿到计算结果并恢复出明文数据。每个参与方可充当多个角色，如一个数据拥有方也可以作为计算方参与训练。
在ABY3[11]中，参与方可分为：输入方、计算方和结果方。输入方为训练数据及模型的持有方，负责加密数据和模型，并将其发送到计算方。计算方为训练的执行方，基于特定的多方安全计算协议完成训练任务。计算方只能得到加密后的数据
及模型，以保证数据隐私。计算结束后，结果方会拿到计算结果并恢复出明文数据。每个参与方可充当多个角色，如一个数据拥有方也可以作为计算方参与训练。
PFM的整个训练及推理过程主要由三个部分组成：数据准备，训练/推理，结果解析。
#### A. 数据准备
##### 1. 私有数据对齐
- **私有数据对齐**： PFM允许数据拥有方（数据方）在不泄露自己数据的情况下，找出多方共有的样本集合。此功能在纵向联邦学习中非常必要，因为其要求多个数据方在训练前进行数据对齐，并且保护用户的数据隐私。
- **数据加密及分发**：在PFM中，数据方将数据和模型用秘密共享[10]的方法加密，然后用直接传输或者数据库存储的方式传到计算方。每个计算方只会拿到数据的一部分，因此计算方无法还原真实数据。
PFM允许数据拥有方（数据方）在不泄露自己数据的情况下，找出多方共有的样本集合。此功能在纵向联邦学习中非常必要，因为其要求多个数据方在训练前进行数据对齐，并且保护用户的数据隐私。凭借PSI算法，PFM可以在一秒内完成6万条数据的对齐。
#### B. 训练/推理
##### 2. 数据加密及分发
PFM 拥有与PaddlePaddle相同的运行模式。在训练前，用户需要定义MPC协议，训练模型以及训练策略。`paddle_fl.mpc`中提供了可以操作加密数据的算子，在运行时算子的实例会被创建并被执行器依次运行。
在PFM中，数据方将数据和模型用秘密共享[10]的方法加密，然后用直接传输或者数据库存储的方式传到计算方。每个计算方只会拿到数据的一部分，因此计算方无法还原真实数据。
#### B. 训练及推理
<img src='images/PFM-design.png' width = "1000" height = "622" align="middle"/>
像PaddlePaddle一样，训练和推理任务可以分为编译阶段和运行阶段。
##### 1. 编译时
* **确定MPC环境**：用户需要指定用到的MPC协议，并配置网络环境。现有版本的Paddle Encrypted只支持"ABY3"协议。更多的协议将在后续版本中支持。
* **用户定义训练任务**：用户可以根据PFM提供的安全接口，定义集齐学习网络以及训练策略。
##### 2. 运行时
一个Paddle Encrypted程序实际上就是一个PaddlePaddle程序。在运行时，PFM的程序将会转变为PaddlePaddle中的ProgramDesc，并交给Executor运行。以下是运行阶段的主要概念：
* **运算节点**：计算节点是与计算方相对应的实体。在实际部署中，它可以是裸机、云虚拟机、docker甚至进程。PFM在每次运行中只需要三个计算节点，这由底层ABY3协议决定。Paddle Encrypted程序将在所有三个计算节点上并行部署和运行。
* **基于MPC的算子**：PFM 为操作加密数据提供了特殊的算子，这些算子在PaddlePaddle框架中实现，基于像ABY3一样的MPC协议。像PaddlePaddle中一样，在运行时PFM的算子将被创建并按照顺序执行。
请参考以下[文档](./docs/source/md/mpc_train_cn.md), 以获得更多关于训练阶段的信息。
#### C. 结果重构
安全训练和推理工作完成后，模型（或预测结果）将由计算方以加密形式输出。结果方可以收集加密的结果，使用PFM中的工具对其进行解密，并将明文结果传递给用户。
请参考[MPC的例子](./python/paddle_fl/mpc/examples)，以获取更多的信息。
## Kubernetes简单部署
### 横向联邦方案
```sh
kubectl apply -f ./python/paddle_fl/paddle_fl/examples/k8s_deployment/master.yaml
```
请参考[K8S部署实例](./python/paddle_fl/paddle_fl/examples/k8s_deployment/README.md)
也可以参考[K8S集群申请及kubectl安装](./python/paddle_fl/paddle_fl/examples/k8s_deployment/deploy_instruction.md) 配置自己的K8S集群
### Federated Learning with MPC
会在后续版本中发布。
## 性能测试
### 横向联邦方案
Gru4Rec [9] 在基于会话的推荐中引入了递归神经网络模型。PaddlePaddle的GRU4RC实现代码在 https://github.com/PaddlePaddle/models/tree/develop/PaddleRec/gru4rec. 一个基于联邦学习训练Gru4Rec模型的示例请参考[Gru4Rec in Federated Learning](https://paddlefl.readthedocs.io/en/latest/examples/gru4rec_examples.html)
### Federated Learning with MPC
#### A. 精度测试
##### 1. 训练参数
- 数据集：波士顿房价预测。
- 训练轮数： 20
- Batch Size：10
##### 2. 实验结果
| Epoch/Step | paddle_fl.mpc | Paddle |
| ---------- | ------------- | ------ |
| Epoch=0, Step=0 | 738.39491 | 738.46204 |
| Epoch=1, Step=0 | 630.68834 | 629.9071 |
| Epoch=2, Step=0 | 539.54683 | 538.1757 |
| Epoch=3, Step=0 | 462.41159 | 460.64722 |
| Epoch=4, Step=0 | 397.11516 | 395.11017 |
| Epoch=5, Step=0 | 341.83102 | 339.69815 |
| Epoch=6, Step=0 | 295.01114 | 292.83597 |
| Epoch=7, Step=0 | 255.35141 | 253.19429 |
| Epoch=8, Step=0 | 221.74739 | 219.65132 |
| Epoch=9, Step=0 | 193.26459 | 191.25981 |
| Epoch=10, Step=0 | 169.11423 | 167.2204 |
| Epoch=11, Step=0 | 148.63138 | 146.85835 |
| Epoch=12, Step=0 | 131.25081 | 129.60391 |
| Epoch=13, Step=0 | 116.49708 | 114.97599 |
| Epoch=14, Step=0 | 103.96669 | 102.56854 |
| Epoch=15, Step=0 | 93.31706 | 92.03858 |
| Epoch=16, Step=0 | 84.26219 | 83.09653 |
| Epoch=17, Step=0 | 76.55664 | 75.49785 |
| Epoch=18, Step=0 | 69.99673 | 69.03561 |
| Epoch=19, Step=0 | 64.40562 | 63.53539 |
我们基于波士顿房价数据集对PFM进行了测试，具体的事例及实现请参考 [uci_demo](./python/paddle_fl/mpc/examples/uci_demo)
## 正在进行的工作
- 纵向联合学习支持更多的模型。
- 发布纵向联邦学习K8S部署方案。
- 手机端的联邦学习模拟器将在下一版本开源。
## 参考文献
......
docs/requirements.txt
浏览文件 @ c069531d
sphinx==2.1.0
mistune
sphinx_rtd_theme
paddlepaddle>=1.6
paddlepaddle>=1.8
zmq
docs/source/compile_and_intall.rst 0 → 100644
浏览文件 @ c069531d
.. mdinclude:: md/compile_and_install.md
docs/source/data_parallel_instruction.rst 0 → 100644
浏览文件 @ c069531d
.. mdinclude:: md/data_parallel_instruction.md
docs/source/examples/dpsgd_examples.rst 0 → 100644
浏览文件 @ c069531d
.. mdinclude:: md/dpsgd-example.md
docs/source/examples/md/dpsgd-example.md
浏览文件 @ c069531d
......@@ -10,7 +10,7 @@ This document introduces how to use PaddleFL to train a model with Fl Strategy:
Please use pip which has paddlepaddle installed
```
```sh
pip install paddle_fl
```
......@@ -35,7 +35,7 @@ The dataset will downloaded automatically in the API and will be located under `
PaddleFL has two phases , CompileTime and RunTime. In CompileTime, a federated learning task is defined by fl_master. In RunTime, a federated learning job is executed on fl_server and fl_trainer in distributed clusters.
```
```sh
sh run.sh
```
......@@ -43,13 +43,19 @@ sh run.sh
In this example, we implement compile time programs in fl_master.py
```
```sh
python fl_master.py
```
In fl_master.py, we first define FL-Strategy, User-Defined-Program and Distributed-Config. Then FL-Job-Generator generate FL-Job for federated server and worker.
```python
import paddle.fluid as fluid
import paddle_fl.paddle_fl as fl
from paddle_fl.paddle_fl.core.master.job_generator import JobGenerator
from paddle_fl.paddle_fl.core.strategy.fl_strategy_base import FLStrategyFactory
import math
class Model(object):
def __init__(self):
pass
......@@ -100,7 +106,7 @@ job_generator.generate_fl_job(
#### How to work in RunTime
```
```sh
python -u fl_scheduler.py >scheduler.log &
python -u fl_server.py >server0.log &
python -u fl_trainer.py 0 >trainer0.log &
......@@ -110,7 +116,9 @@ python -u fl_trainer.py 3 >trainer3.log &
```
In fl_scheduler.py, we let server and trainers to do registeration.
```
```python
from paddle_fl.paddle_fl.core.scheduler.agent_master import FLScheduler
worker_num = 4
server_num = 1
#Define number of worker/server and the port for scheduler
......@@ -122,7 +130,12 @@ scheduler.start_fl_training()
```
In fl_server.py, we load and run the FL server job.
```
```python
import paddle_fl.paddle_fl as fl
import paddle.fluid as fluid
from paddle_fl.paddle_fl.core.server.fl_server import FLServer
from paddle_fl.paddle_fl.core.master.fl_job import FLRunTimeJob
server = FLServer()
server_id = 0
job_path = "fl_job_config"
......@@ -136,18 +149,23 @@ server.start()
In fl_trainer.py, we load and run the FL trainer job, then evaluate the accuracy with test data and compute the privacy budget.
```
```python
import numpy
import sys
import paddle
import paddle.fluid as fluid
import logging
import math
from paddle_fl.paddle_fl.core.master.fl_job import FLRunTimeJob
from paddle_fl.paddle_fl.core.trainer.fl_trainer import FLTrainerFactory
trainer_id = int(sys.argv[1]) # trainer id for each guest
job_path = "fl_job_config"
job = FLRunTimeJob()
job.load_trainer_job(job_path, trainer_id)
trainer = FLTrainerFactory().create_fl_trainer(job)
trainer.start()
```
```
def train_test(train_test_program, train_test_feed, train_test_reader):
acc_set = []
for test_data in train_test_reader():
......
docs/source/examples/md/gru4rec_examples.md
浏览文件 @ c069531d
......@@ -39,6 +39,11 @@ python fl_master.py
```
In fl_master.py, we first define FL-Strategy, User-Defined-Program and Distributed-Config. Then FL-Job-Generator generate FL-Job for federated server and worker.
```python
import paddle.fluid as fluid
import paddle_fl.paddle_fl as fl
from paddle_fl.paddle_fl.core.master.job_generator import JobGenerator
from paddle_fl.paddle_fl.core.strategy.fl_strategy_base import FLStrategyFactory
# define model
model = Model()
model.gru4rec_network()
......
docs/source/examples/md/uci_demo.md 0 → 100644
浏览文件 @ c069531d
## Instructions for PaddleFL-MPC UCI Housing Demo
([简体中文](./README_CN.md)|English)
This document introduces how to run UCI Housing demo based on Paddle-MPC, which has two ways of running, i.e., single machine and multi machines.
### 1. Running on Single Machine
#### (1). Prepare Data
Generate encrypted data utilizing `generate_encrypted_data()` in `process_data.py` script. For example, users can write the following code into a python script named `prepare.py`, and then run the script with command `python prepare.py`.
```python
import process_data
process_data.generate_encrypted_data()
```
Encrypted data files of feature and label would be generated and saved in `/tmp` directory. Different suffix names are used for these files to indicate the ownership of different computation parties. For instance, a file named `house_feature.part0` means it is a feature file of party 0.
#### (2). Launch Demo with A Shell Script
Launch demo with the `run_standalone.sh` script. The concrete command is:
```bash
bash run_standalone.sh uci_housing_demo.py
```
The loss with cypher text format will be displayed on screen while training. At the same time, the loss data would be also save in `/tmp` directory, and the format of file name is similar to what is described in Step 1.
Besides, predictions would be made in this demo once training is finished. The predictions with cypher text format would also be save in `/tmp` directory.
Finally, using `load_decrypt_data()` in `process_data.py` script, this demo would decrypt and print the loss and predictions, which can be compared with related results of Paddle plain text model.
**Note** that remember to delete the loss and prediction files in `/tmp` directory generated in last running, in case of any influence on the decrypted results of current running. For simplifying users operations, we provide the following commands in `run_standalone.sh`, which can delete the files mentioned above when running this script.
```bash
# remove temp data generated in last time
LOSS_FILE="/tmp/uci_loss.*"
PRED_FILE="/tmp/uci_prediction.*"
if [ "$LOSS_FILE" ]; then
rm -rf $LOSS_FILE
fi
if [ "$PRED_FILE" ]; then
rm -rf $PRED_FILE
fi
```
### 2. Running on Multi Machines
#### (1). Prepare Data
Data owner encrypts data. Concrete operations are consistent with “Prepare Data” in “Running on Single Machine”.
#### (2). Distribute Encrypted Data
According to the suffix of file name, distribute encrypted data files to `/tmp ` directories of all 3 computation parties. For example, send `house_feature.part0` and `house_label.part0` to `/tmp` of party 0 with `scp` command.
#### (3). Modify uci_housing_demo.py
Each computation party makes the following modifications on `uci_housing_demo.py` according to the environment of machine.
* Modify IP Information
Modify `localhost` in the following code as the IP address of the machine.
```python
pfl_mpc.init("aby3", int(role), "localhost", server, int(port))
```
* Comment Out Codes for Single Machine Running
Comment out the following codes which are used when running on single machine.
```python
import process_data
print("uci_loss:")
process_data.load_decrypt_data("/tmp/uci_loss", (1,))
print("prediction:")
process_data.load_decrypt_data("/tmp/uci_prediction", (BATCH_SIZE,))
```
#### (4). Launch Demo on Each Party
**Note** that Redis service is necessary for demo running. Remember to clear the cache of Redis server before launching demo on each computation party, in order to avoid any negative influences caused by the cached records in Redis. The following command can be used for clear Redis, where REDIS_BIN is the executable binary of redis-cli, SERVER and PORT represent the IP and port of Redis server respectively.
```
$REDIS_BIN -h $SERVER -p $PORT flushall
```
Launch demo on each computation party with the following command,
```
$PYTHON_EXECUTABLE uci_housing_demo.py $PARTY_ID $SERVER $PORT
```
where PYTHON_EXECUTABLE is the python which installs PaddleFL, PARTY_ID is the ID of computation party, which is 0, 1, or 2, SERVER and PORT represent the IP and port of Redis server respectively.
Similarly, training loss with cypher text format would be printed on the screen of each computation party. And at the same time, the loss and predictions would be saved in `/tmp` directory.
**Note** that remember to delete the loss and prediction files in `/tmp` directory generated in last running, in case of any influence on the decrypted results of current running.
#### (5). Decrypt Loss and Prediction Data
Each computation party sends `uci_loss.part` and `uci_prediction.part` files in `/tmp` directory to the `/tmp` directory of data owner. Data owner decrypts and gets the plain text of loss and predictions with ` load_decrypt_data()` in `process_data.py`.
For example, the following code can be written into a python script to decrypt and print training loss.
```python
import process_data
print("uci_loss:")
process_data.load_decrypt_data("/tmp/uci_loss", (1,))
```
And the following code can be written into a python script to decrypt and print predictions.
```python
import process_data
print("prediction:")
process_data.load_decrypt_data("/tmp/uci_prediction", (BATCH_SIZE,))
```
#### Convergence of paddle_fl.mpc vs paddle
Below, is the result of our experiment to test the convergence of paddle_fl.mpc
#### A. Convergence of paddle_fl.mpc vs paddle
##### 1. Training Parameters
- Dataset: Boston house price dataset
- Number of Epoch: 20
- Batch Size: 10
##### 2. Experiment Results
| Epoch/Step | paddle_fl.mpc | Paddle |
| ---------- | ------------- | ------ |
| Epoch=0, Step=0 | 738.39491 | 738.46204 |
| Epoch=1, Step=0 | 630.68834 | 629.9071 |
| Epoch=2, Step=0 | 539.54683 | 538.1757 |
| Epoch=3, Step=0 | 462.41159 | 460.64722 |
| Epoch=4, Step=0 | 397.11516 | 395.11017 |
| Epoch=5, Step=0 | 341.83102 | 339.69815 |
| Epoch=6, Step=0 | 295.01114 | 292.83597 |
| Epoch=7, Step=0 | 255.35141 | 253.19429 |
| Epoch=8, Step=0 | 221.74739 | 219.65132 |
| Epoch=9, Step=0 | 193.26459 | 191.25981 |
| Epoch=10, Step=0 | 169.11423 | 167.2204 |
| Epoch=11, Step=0 | 148.63138 | 146.85835 |
| Epoch=12, Step=0 | 131.25081 | 129.60391 |
| Epoch=13, Step=0 | 116.49708 | 114.97599 |
| Epoch=14, Step=0 | 103.96669 | 102.56854 |
| Epoch=15, Step=0 | 93.31706 | 92.03858 |
| Epoch=16, Step=0 | 84.26219 | 83.09653 |
| Epoch=17, Step=0 | 76.55664 | 75.49785 |
| Epoch=18, Step=0 | 69.99673 | 69.03561 |
| Epoch=19, Step=0 | 64.40562 | 63.53539 |
docs/source/examples/mpc_uci_demo.rst 0 → 100644
浏览文件 @ c069531d
.. mdinclude:: md/uci_demo.md
docs/source/index.rst
浏览文件 @ c069531d
......@@ -2,19 +2,13 @@
.. mdinclude:: md/logo.md
Quick Start
===========
.. toctree::
:maxdepth: 1
:caption: Quick Start
:hidden:
instruction.rst
See instruction_ for quick start.
.. _instruction: instruction.html
compile_and_intall.rst
data_parallel_instruction.rst
mpc_instruction.rst
.. toctree::
:maxdepth: 1
......@@ -31,14 +25,8 @@ See instruction_ for quick start.
:caption: Examples
examples/gru4rec_examples.rst
.. toctree::
:maxdepth: 2
:caption: API Reference
api/paddle_fl
examples/dpsgd_examples.rst
examples/mpc_uci_demo.rst
The Team
========
......@@ -49,7 +37,7 @@ The Team
team.rst
PaddleFL is developed and maintained by Nimitz Team at Baidu
PaddleFL is developed by PaddlePaddle and Security team.
.. toctree::
......
docs/source/instruction.rst 已删除 100644 → 0
浏览文件 @ 7d144c3b
Quick Start Instructions
========================
.. mdinclude:: md/logo.md
Install PaddleFL
----------------
To install PaddleFL, we need the following packages.
.. code-block:: sh
paddlepaddle >= 1.6
networkx
We can run
.. code-block:: sh
python setup.py install
or
pip install paddle-fl
.. mdinclude:: md/quick_start.md
docs/source/md/compile_and_install.md 0 → 100644
浏览文件 @ c069531d
# Compile and Install
## Installation
We **highly recommend** to run PaddleFL in Docker
```sh
#Pull and run the docker
docker pull hub.baidubce.com/paddlefl/paddle_fl:latest
docker run --name <docker_name> --net=host -it -v $PWD:/root <image id> /bin/bash
#Install paddle_fl
pip install paddle_fl
```
We also prepare a stable redis package for you to download and install, which will be used in tasks with MPC.
```sh
wget --no-check-certificate https://paddlefl.bj.bcebos.com/redis-stable.tar
tar -xf redis-stable.tar
cd redis-stable && make
```
## Compile From Source Code
#### A. Environment preparation
* CentOS 6 or CentOS 7 (64 bit)
* Python 2.7.15+/3.5.1+/3.6/3.7 ( 64 bit) or above
* pip or pip3 9.0.1+ (64 bit)
* PaddlePaddle release 1.8
* Redis 5.0.8 (64 bit)
* GCC or G++ 4.8.3+
* cmake 3.15+
#### B. Clone the source code, compile and install
Fetch the source code and checkout stable release
```sh
git clone https://github.com/PaddlePaddle/PaddleFL
cd /path/to/PaddleFL
# Checkout stable release
mkdir build && cd build
```
Execute compile commands, where `PYTHON_EXECUTABLE` is path to the python binary where the PaddlePaddle is installed, `CMAKE_CXX_COMPILER` is the path of G++ and `PYTHON_INCLUDE_DIRS` is the corresponding python include directory. You can get the `PYTHON_INCLUDE_DIRS` via the following command:
```sh
${PYTHON_EXECUTABLE} -c "from distutils.sysconfig import get_python_inc;print(get_python_inc())"
```
Then you can put the directory in the following command and make:
```sh
cmake ../ -DPYTHON_EXECUTABLE=${PYTHON_EXECUTABLE} -DPYTHON_INCLUDE_DIRS=${python_include_dir} -DCMAKE_CXX_COMPILER=${g++_path}
make -j$(nproc)
```
Install the package:
```sh
make install
cd /path/to/PaddleFL/python
${PYTHON_EXECUTABLE} setup.py sdist bdist_wheel
pip or pip3 install dist/***.whl -U
```
docs/source/md/compile_and_install_cn.md 0 → 100644
浏览文件 @ c069531d
## 从源码编译
#### A. 环境准备
* CentOS 6 or CentOS 7 (64 bit)
* Python 2.7.15+/3.5.1+/3.6/3.7 ( 64 bit) or above
* pip or pip3 9.0.1+ (64 bit)
* PaddlePaddle release 1.8
* Redis 5.0.8 (64 bit)
* GCC or G++ 4.8.3+
* cmake 3.15+
#### B. 克隆源代码并编译安装
获取源代码
```sh
git clone https://github.com/PaddlePaddle/PaddleFL
cd /path/to/PaddleFL
# Checkout stable release
mkdir build && cd build
```
执行编译指令, `PYTHON_EXECUTABLE` 为安装了PaddlePaddle的可执行python路径, `CMAKE_CXX_COMPILER` 为指定的g++路径。 `PYTHON_INCLUDE_DIRS` 是相应的include路径，可以用如下指令获得：
```sh
${PYTHON_EXECUTABLE} -c "from distutils.sysconfig import get_python_inc;print(get_python_inc())"
```
之后就可以执行编译和安装的指令
```sh
cmake ../ -DPYTHON_EXECUTABLE=${PYTHON_EXECUTABLE} -DPYTHON_INCLUDE_DIRS=${python_include_dir} -DCMAKE_CXX_COMPILER=${g++_path}
make -j$(nproc)
```
安装对应的安装包
```sh
make install
cd /path/to/PaddleFL/python
${PYTHON_EXECUTABLE} setup.py sdist bdist_wheel
pip or pip3 install dist/***.whl -U
```
docs/source/md/quick_start.md → docs/source/md/data_parallel_instruction.md
浏览文件 @ c069531d
# Instructions for Data Parallel
## Step 1: Define Federated Learning Compile-Time
We define very simple multiple layer perceptron for demonstration. When multiple organizations
......
docs/source/md/introduction.md
浏览文件 @ c069531d
......@@ -8,15 +8,17 @@ Data is becoming more and more expensive nowadays, and sharing of raw data is ve
## Overview of PaddleFL
<img src='_static/FL-framework.png' width = "1300" height = "310" align="middle"/>
<img src='../../../images/FL-framework.png' width = "1000" height = "320" align="middle"/>
In PaddleFL, horizontal and vertical federated learning strategies will be implemented according to the categorization given in [4]. Application demonstrations in natural language processing, computer vision and recommendation will be provided in PaddleFL.
#### Federated Learning Strategy
- **Vertical Federated Learning**: Logistic Regression with PrivC, Neural Network with third-party PrivC [5]
#### A. Federated Learning Strategy
- **Vertical Federated Learning**: Logistic Regression with PrivC[5], Neural Network with MPC [11]
- **Horizontal Federated Learning**: Federated Averaging [2], Differential Privacy [6]
- **Horizontal Federated Learning**: Federated Averaging [2], Differential Privacy [6], Secure Aggregation
#### Training Strategy
#### B. Training Strategy
- **Multi Task Learning** [7]
......@@ -24,15 +26,23 @@ In PaddleFL, horizontal and vertical federated learning strategies will be imple
- **Active Learning**
There are mainly two components in PaddleFL: **Data Parallel** and **Federated Learning with MPC (PFM)**.
With Data Parallel, distributed data holders can finish their Federated Learning tasks based on common horizontal federated strategies, such as FedAvg, DPSGD, etc.
Besides, PFM is implemented based on secure multi-party computation (MPC) to enable secure training and prediction. As a key product of PaddleFL, PFM intrinsically supports federated learning well, including horizontal, vertical and transfer learning scenarios. Users with little cryptography expertise can also train models or conduct prediction on encrypted data.
## Framework design of PaddleFL
<img src='_static/FL-training.png' width = "1300" height = "400" align="middle"/>
### Data Parallel
<img src='images/FL-training.png' width = "1000" height = "400" align="middle"/>
In PaddleFL, components for defining a federated learning task and training a federated learning job are as follows:
In Data Parallel, components for defining a federated learning task and training a federated learning job are as follows:
#### Compile Time
#### A. Compile Time
- **FL-Strategy**: a user can define federated learning strategies with FL-Strategy such as Fed-Avg[1]
- **FL-Strategy**: a user can define federated learning strategies with FL-Strategy such as Fed-Avg[2]
- **User-Defined-Program**: PaddlePaddle's program that defines the machine learning model structure and training strategies such as multi-task learning.
......@@ -40,7 +50,7 @@ In PaddleFL, components for defining a federated learning task and training a fe
- **FL-Job-Generator**: Given FL-Strategy, User-Defined Program and Distributed Training Config, FL-Job for federated server and worker will be generated through FL Job Generator. FL-Jobs will be sent to organizations and federated parameter server for run-time execution.
#### Run Time
#### B. Run Time
- **FL-Server**: federated parameter server that usually runs in cloud or third-party clusters.
......@@ -48,10 +58,37 @@ In PaddleFL, components for defining a federated learning task and training a fe
- **FL-scheduler**: Decide which set of trainers can join the training before each updating cycle.
## On Going and Future Work
### Federated Learning with MPC
<img src='../../../images/PFM-overview.png' width = "1000" height = "446" align="middle"/>
Paddle FL MPC implements secure training and inference tasks based on the underlying MPC protocol like ABY3[11], which is a high efficient three-party computing model.
In ABY3, participants can be classified into roles of Input Party (IP), Computing Party (CP) and Result Party (RP). Input Parties (e.g., the training data/model owners) encrypt and distribute data or models to Computing Parties. Computing Parties (e.g., the VM on the cloud) conduct training or inference tasks based on specific MPC protocols, being restricted to see only the encrypted data or models, and thus guarantee the data privacy. When the computation is completed, one or more Result Parties (e.g., data owners or specified third-party) receive the encrypted results from Computing Parties, and reconstruct the plaintext results. Roles can be overlapped, e.g., a data owner can also act as a computing party.
A full training or inference process in PFM consists of mainly three phases: data preparation, training/inference, and result reconstruction.
#### A. Data preparation
- **Private data alignment**: PFM enables data owners (IPs) to find out records with identical keys (like UUID) without revealing private data to each other. This is especially useful in the vertical learning cases where segmented features with same keys need to be identified and aligned from all owners in a private manner before training.
- **Encryption and distribution**: In PFM, data and models from IPs will be encrypted using Secret-Sharing[10], and then be sent to CPs, via directly transmission or distributed storage like HDFS. Each CP can only obtain one share of each piece of data, and thus is unable to recover the original value in the Semi-honest model.
#### B. Training/inference
A PFM program is exactly a PaddlePaddle program, and will be executed as normal PaddlePaddle programs. Before training/inference, user needs to choose a MPC protocol, define a machine learning model and their training strategies. Typical machine learning operators are provided in `paddle_fl.mpc` over encrypted data, of which the instances are created and run in order by Executor during run-time.
#### C. Result reconstruction
Upon completion of the secure training (or inference) job, the models (or prediction results) will be output by CPs in encrypted form. Result Parties can collect the encrypted results, decrypt them using the tools in PFM, and deliver the plaintext results to users.
# On Going and Future Work
- Vertial Federated Learning will support more algorithms.
- Add K8S deployment scheme for Paddle Encrypted.
- Experimental benchmark with public datasets in federated learning settings.
- FL mobile simulator will be open sourced in following versions.
- Federated Learning Systems deployment methods in Kubernetes.
- Vertical Federated Learning Strategies and more horizontal federated learning strategies will be open sourced.
docs/source/md/mpc_instruction.md 0 → 100644
浏览文件 @ c069531d
# Instructions for Federated Learning with MPC
A PFM program is exactly a PaddlePaddle program, and will be executed as normal PaddlePaddle programs. Before training/inference, user needs to choose a MPC protocol, define a machine learning model and their training strategies. Typical machine learning operators are provided in paddle_fl.mpc over encrypted data, of which the instances are created and run in order by Executor during run-time.
Below is an example for complish an vertial Federated Learning with MPC
```python
import sys
import numpy as np
import time
import paddle
import paddle.fluid as fluid
import paddle_fl.mpc as pfl_mpc
import paddle_fl.mpc.data_utils.aby3 as aby3
# define your role number(0, 1, 2) and the address of redis server
role, server, port = sys.argv[1], sys.argv[2], sys.argv[3]
# specify the protocal and initialize the environment
pfl_mpc.init("aby3", int(role), "localhost", server, int(port))
role = int(role)
# data preprocessing
BATCH_SIZE = 10
feature_reader = aby3.load_aby3_shares("/tmp/house_feature", id=role, shape=(13, ))
label_reader = aby3.load_aby3_shares("/tmp/house_label", id=role, shape=(1, ))
batch_feature = aby3.batch(feature_reader, BATCH_SIZE, drop_last=True)
batch_label = aby3.batch(label_reader, BATCH_SIZE, drop_last=True)
x = pfl_mpc.data(name='x', shape=[BATCH_SIZE, 13], dtype='int64')
y = pfl_mpc.data(name='y', shape=[BATCH_SIZE, 1], dtype='int64')
# async data loader
loader = fluid.io.DataLoader.from_generator(feed_list=[x, y], capacity=BATCH_SIZE)
batch_sample = paddle.reader.compose(batch_feature, batch_label)
place = fluid.CPUPlace()
loader.set_batch_generator(batch_sample, places=place)
# define your model
y_pre = pfl_mpc.layers.fc(input=x, size=1)
infer_program = fluid.default_main_program().clone(for_test=False)
cost = pfl_mpc.layers.square_error_cost(input=y_pre, label=y)
avg_loss = pfl_mpc.layers.mean(cost)
optimizer = pfl_mpc.optimizer.SGD(learning_rate=0.001)
optimizer.minimize(avg_loss)
# give the path to store training loss
loss_file = "/tmp/uci_loss.part{}".format(role)
# train
exe = fluid.Executor(place)
exe.run(fluid.default_startup_program())
epoch_num = 20
start_time = time.time()
for epoch_id in range(epoch_num):
step = 0
# Method 1: feed data directly
# for feature, label in zip(batch_feature(), batch_label()):
# mpc_loss = exe.run(feed={"x": feature, "y": label}, fetch_list=[avg_loss])
# Method 2: feed data via loader
for sample in loader():
mpc_loss = exe.run(feed=sample, fetch_list=[avg_loss])
if step % 50 == 0:
print('Epoch={}, Step={}, Loss={}'.format(epoch_id, step,
mpc_loss))
with open(loss_file, 'ab') as f:
f.write(np.array(mpc_loss).tostring())
step += 1
end_time = time.time()
print('Mpc Training of Epoch={} Batch_size={}, cost time in seconds:{}'
.format(epoch_num, BATCH_SIZE, (end_time - start_time)))
```
docs/source/md/mpc_train.md 0 → 100644
浏览文件 @ c069531d
## Training/inference in MPC
<img src='../../../images/PFM-design.png' width = "1000" height = "622" align="middle"/>
As in PaddlePaddle, a training or inference job can be separated into the compile-time phase and the run-time phase:
#### Compile time
* **MPC environment specification**: a user needs to choose a MPC protocol, and configure the network settings. In current version, PFM provides only the "ABY3" protocol. More protocol implementation will be provided in future.
* **User-defined job program**: a user can define the machine learning model structure and the training strategies (or inference task) in a PFM program, using the secure operators.
#### Run time
A PFM program is exactly a PaddlePaddle program, and will be executed as normal PaddlePaddle programs. For example, in run-time a PFM program will be transpiled into ProgramDesc, and then be passed to and run by the Executor. The main concepts in the run-time phase are as follows:
* **Computing nodes**: a computing node is an entity corresponding to a Computing Party. In real deployment, it can be a bare-metal machine, a cloud VM, a docker or even a process. PFM requires exactly three computing nodes in each run, which is determined by the underlying ABY3 protocol. A PFM program will be deployed and run in parallel on all three computing nodes.
* **Operators using MPC**: PFM provides typical machine learning operators in `paddle_fl.mpc` over encrypted data. Such operators are implemented upon PaddlePaddle framework, based on MPC protocols like ABY3. Like other PaddlePaddle operators, in run time, instances of PFM operators are created and run in order by Executor.
docs/source/md/mpc_train_cn.md 0 → 100644
浏览文件 @ c069531d
## MPC 训练及推理
<img src='../../../images/PFM-design.png' width = "1000" height = "622" align="middle"/>
像PaddlePaddle一样，训练和推理任务可以分为编译阶段和运行阶段。
##### 1. 编译时
* **确定MPC环境**：用户需要指定用到的MPC协议，并配置网络环境。现有版本的Paddle Encrypted只支持"ABY3"协议。更多的协议将在后续版本中支持。
* **用户定义训练任务**：用户可以根据PFM提供的安全接口，定义集齐学习网络以及训练策略。
##### 2. 运行时
一个Paddle Encrypted程序实际上就是一个PaddlePaddle程序。在运行时，PFM的程序将会转变为PaddlePaddle中的ProgramDesc，并交给Executor运行。以下是运行阶段的主要概念：
* **运算节点**：计算节点是与计算方相对应的实体。在实际部署中，它可以是裸机、云虚拟机、docker甚至进程。PFM在每次运行中只需要三个计算节点，这由底层ABY3协议决定。Paddle Encrypted程序将在所有三个计算节点上并行部署和>运行。
* **基于MPC的算子**：PFM 为操作加密数据提供了特殊的算子，这些算子在PaddlePaddle框架中实现，基于像ABY3一样的MPC协议。像PaddlePaddle中一样，在运行时PFM的算子将被创建并按照顺序执行。
docs/source/md/reference.md
浏览文件 @ c069531d
## Reference
[1]. Jakub Konen, H. Brendan McMahan, Daniel Ramage, Peter Richtik. **Federated Optimization: Distributed Machine Learning for On-Device Intelligence.** 2016
[1]. Jakub Konečný, H. Brendan McMahan, Daniel Ramage, Peter Richtárik. **Federated Optimization: Distributed Machine Learning for On-Device Intelligence.** 2016
[2]. H. Brendan McMahan, Eider Moore, Daniel Ramage, Blaise Agera y Arcas. **Federated Learning of Deep Networks using Model Averaging.** 2017
[2]. H. Brendan McMahan, Eider Moore, Daniel Ramage, Blaise Agüera y Arcas. **Federated Learning of Deep Networks using Model Averaging.** 2017
[3]. Jakub Konen, H. Brendan McMahan, Felix X. Yu, Peter Richtik, Ananda Theertha Suresh, Davepen Bacon. **Federated Learning: Strategies for Improving Communication Efficiency.** 2016
[3]. Jakub Konečný, H. Brendan McMahan, Felix X. Yu, Peter Richtárik, Ananda Theertha Suresh, Dave Bacon. **Federated Learning: Strategies for Improving Communication Efficiency.** 2016
[4]. Qiang Yang, Yang Liu, Tianjian Chen, Yongxin Tong. **Federated Machine Learning: Concept and Applications.** 2019
[5]. Kai He, Liu Yang, Jue Hong, Jinghua Jiang, Jieming Wu, Xu Dong et al. **PrivC - A framework for efficient Secure Two-Party Computation. In Proceedings of 15th EAI International Conference on Security and Privacy in Communication Networks.** SecureComm 2019
[5]. Kai He, Liu Yang, Jue Hong, Jinghua Jiang, Jieming Wu, Xu Dong et al. **PrivC - A framework for efficient Secure Two-Party Computation.** In Proc. of SecureComm 2019
[6]. Mart Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, Li Zhang. **Deep Learning with Differential Privacy.** 2016
[6]. Martín Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, Li Zhang. **Deep Learning with Differential Privacy.** 2016
[7]. Virginia Smith, Chao-Kai Chiang, Maziar Sanjabi, Ameet Talwalkar. **Federated Multi-Task Learning** 2016
[8]. Yang Liu, Tianjian Chen, Qiang Yang. **Secure Federated Transfer Learning.** 2018
[9]. Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, Domonkos Tikk. **Session-based Recommendations with Recurrent Neural Networks.** 2016
[10]. https://en.wikipedia.org/wiki/Secret_sharing
[11]. Payman Mohassel and Peter Rindal. **ABY3: A Mixed Protocol Framework for Machine Learning.** In Proc. of CCS 2018
docs/source/mpc_instruction.rst 0 → 100644
浏览文件 @ c069531d
.. mdinclude:: md/mpc_instruction.md
docs/source/team.rst
浏览文件 @ c069531d
The Team
========
PGL is developed and maintained by NLP and Paddle Teams at Baidu
PaddleFL is developed by PaddlePaddle and Security team.
Markdown is supported
0% 或 .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑！
想要评论请 注册 或