The LeNet model and MNIST dataset are used as an example to describe how to use the differential privacy optimizer to train a neural network model on MindSpore.
> This example is for the Ascend 910 AI processor and supports PYNATIVE_MODE. You can download the complete sample code from <https://gitee.com/mindspore/mindarmour/blob/master/example/mnist_demo/lenet5_dp_model_train.py>.
> This example is for the Ascend 910 AI processor. You can download the complete sample code from <https://gitee.com/mindspore/mindarmour/blob/master/example/mnist_demo/lenet5_dp.py>.
## Implementation
...
...
@@ -82,12 +82,12 @@ TAG = 'Lenet5_train'
### Configuring Parameters
1. Set the running environment, dataset path, model training parameters, checkpoint storage parameters, and differential privacy parameters.
1. Set the running environment, dataset path, model training parameters, checkpoint storage parameters, and differential privacy parameters. Replace 'data_path' with you data path.
```python
cfg=edict({
'num_classes':10,# the number of classes of model's output
'lr':0.1,# the learning rate of model's optimizer
'lr':0.01,# the learning rate of model's optimizer
'momentum':0.9,# the momentum value of model's optimizer
'epoch_size':10,# training epochs
'batch_size':256,# batch size for training
...
...
@@ -99,10 +99,15 @@ TAG = 'Lenet5_train'
'data_path':'./MNIST_unzip',# the path of training and testing data set
'dataset_sink_mode':False,# whether deliver all training data to device one time
'micro_batches':16,# the number of small batches split from an original batch
'norm_clip':1.0,# the clip bound of the gradients of model's training parameters
'initial_noise_multiplier':1.5,# the initial multiplication coefficient of the noise added to training
'norm_bound':1.0,# the clip bound of the gradients of model's training parameters
'initial_noise_multiplier':1.0,# the initial multiplication coefficient of the noise added to training
# parameters' gradients
'mechanisms':'AdaGaussian',# the method of adding noise in gradients while training
'noise_mechanisms':'Gaussian',# the method of adding noise in gradients while training
'clip_mechanisms':'Gaussian',# the method of adaptive clipping gradients while training
'clip_decay_policy':'Linear',# Decay policy of adaptive clipping, decay_policy must be in ['Linear', 'Geometric'].
'clip_learning_rate':0.001,# Learning rate of update norm clip.
'target_unclipped_quantile':0.9,# Target quantile of norm clip.
'fraction_stddev':0.01,# The stddev of Gaussian normal which used in empirical_fraction.
'optimizer':'Momentum'# the base optimizer used for Differential privacy training
})
```
...
...
@@ -110,7 +115,7 @@ TAG = 'Lenet5_train'
2. Configure necessary information, including the environment information and execution mode.
1. Set parameters of a differential privacy optimizer.
- Determine whether values of the **micro_batches** and **batch_size** parameters meet the requirements. The value of **batch_size** must be an integer multiple of **micro_batches**.
- Determine whether values of the `micro_batches` and `batch_size` parameters meet the requirements. The value of `batch_size` must be an integer multiple of `micro_batches`.
- Instantiate a differential privacy factory class.
- Set a noise mechanism for the differential privacy. Currently, the Gaussian noise mechanism with a fixed standard deviation (`Gaussian`) and the Gaussian noise mechanism with an adaptive standard deviation (`AdaGaussian`) are supported.
- Set an optimizer type. Currently, `SGD`, `Momentum`, and `Adam` are supported.
- Set up a differential privacy budget monitor RDP to observe changes in the differential privacy budget $\epsilon$ in each step.
In the preceding command, replace `lenet5_dp_model_train.py` with the name of your script.
In the preceding command, replace `lenet5_dp.py` with the name of your script.
5. Display the result.
The accuracy of the LeNet model without differential privacy is 99%, and the accuracy of the LeNet model with adaptive differential privacy AdaDP is 98%.
The accuracy of the LeNet model without differential privacy is 99%, and the accuracy of the LeNet model with Gaussian noise and adaptive clip differential privacy is 97%.