Search Space used in neural architecture search. Search Space is a collection of model architecture, the purpose of SANAS is to get a model which FLOPs or latency is smaller or percision is higher.
search space which paddleslim.nas provided
-------
Based on origin model architecture:
1. MobileNetV2Space<br>
  MobileNetV2's architecture can reference: [code](https://github.com/PaddlePaddle/models/blob/develop/PaddleCV/image_classification/models/mobilenet_v2.py#L29), [paper](https://arxiv.org/abs/1801.04381)
2. MobileNetV1Space<br>
  MobilNetV1's architecture can reference: [code](https://github.com/PaddlePaddle/models/blob/develop/PaddleCV/image_classification/models/mobilenet_v1.py#L29), [paper](https://arxiv.org/abs/1704.04861)
3. ResNetSpace<br>
  ResNetSpace's architecture can reference: [code](https://github.com/PaddlePaddle/models/blob/develop/PaddleCV/image_classification/models/resnet.py#L30), [paper](https://arxiv.org/pdf/1512.03385.pdf)
Based on block from different model:
1. MobileNetV1BlockSpace<br>
  MobileNetV1Block's architecture can reference: [code](https://github.com/PaddlePaddle/models/blob/develop/PaddleCV/image_classification/models/mobilenet_v1.py#L173)
2. MobileNetV2BlockSpace<br>
  MobileNetV2Block's architecture can reference: [code](https://github.com/PaddlePaddle/models/blob/develop/PaddleCV/image_classification/models/mobilenet_v2.py#L174)
3. ResNetBlockSpace<br>
  ResNetBlock's architecture can reference: [code](https://github.com/PaddlePaddle/models/blob/develop/PaddleCV/image_classification/models/resnet.py#L148)
4. InceptionABlockSpace<br>
  InceptionABlock's architecture can reference: [code](https://github.com/PaddlePaddle/models/blob/develop/PaddleCV/image_classification/models/inception_v4.py#L140)
5. InceptionCBlockSpace<br>
  InceptionCBlock's architecture can reference: [code](https://github.com/PaddlePaddle/models/blob/develop/PaddleCV/image_classification/models/inception_v4.py#L291)
How to use search space
--------
1. Only need to specify the name of search space if use the space based on origin model architecture, such as configs for class SANAS is [('MobileNetV2Space')] if you want to use origin MobileNetV2 as search space.
2. Use search space paddleslim.nas provided based on block:<br>
2.1 Use `input_size`, `output_size` and `block_num` to construct search space, such as configs for class SANAS is ('MobileNetV2BlockSpace', {'input_size': 224, 'output_size': 32, 'block_num': 10})].<br>
2.2 Use `block_mask` to construct search space, such as configs for class SANAS is [('MobileNetV2BlockSpace', {'block_mask': [0, 1, 1, 1, 1, 0, 1, 0]})].
How to write yourself search space
--------
If you want to write yourself search space, you need to inherit base class named SearchSpaceBase and overwrite following functions:<br>
  1. Function to get initial tokens(function `init_tokens`), set the initial tokens which you want, every token in tokens means index of search list, such as if tokens=[0, 3, 5], it means the list of channel of current model architecture is [8, 40, 128].
  2. Function about the length of every token in tokens(function `range_table`), range of every token in tokens.
  3. Function to get model architecture according to tokens(function `token2arch`), get model architecture according to tokens in the search process.
For example, how to add a search space with resnet block. New search space can NOT has the same name with existing search space.
```python
### import necessary head file
from .search_space_base import SearchSpaceBase
from .search_space_registry import SEARCHSPACE
import numpy as np
### use decorator SEARCHSPACE.register to register yourself search space to search space NameSpace
@SEARCHSPACE.register
### define a search space class inherit the base class SearchSpaceBase
# Nerual Architecture Search for Image Classification
This tutorial shows how to use [API](../api/nas_api.md) about SANAS in PaddleSlim. We start experiment based on MobileNetV2 as example. The tutorial contains follow section.
1. necessary imports
2. initial SANAS instance
3. define function about building program
4. define function about input data
5. define function about training
6. define funciton about evaluation
7. start search
7.1 fetch model architecture
7.2 build program
7.3 define input data
7.4 train model
7.5 evaluate model
7.6 reture score
8. full example
The following chapter describes each steps in order.
## 1. import dependency
Please make sure that you haved installed Paddle correctly, then do the necessary imports.
The following steps describes how to get current model architecture and what need to do after get the model architecture. If you want to start a full example directly, please jump to Step 9.
### 7.1 fetch model architecture
Get Next model architecture by `next_archs()`.
```python
archs=sanas.next_archs()[0]
```
### 7.2 build program
Get program according to the function in Step3 and model architecture from Step 7.1.
The following is a full example about neural architecture search, it uses FLOPs as constraint and includes 3 steps, it means train 3 model architectures which is satisfied constraint, and train 7 epoch for each model architecture.
SANAS(Simulated Annealing Neural Architecture Search) is a neural architecture search algorithm
based on simulated annealing, used in discrete search task generally.
Args:
configs(list<tuple>): A list of search space configuration with format [(key, {input_size,
output_size, block_num, block_mask})]. `key` is the name of search space
with data type str. `input_size` and `output_size` are input size and
output size of searched sub-network. `block_num` is the number of blocks
in searched network, `block_mask` is a list consists by 0 and 1, 0 means
normal block, 1 means reduction block.
server_addr(tuple): Server address, including ip and port of server. If ip is None or "", will
use host ip if is_server = True. Default: ("", 8881).
init_temperature(float): Initial temperature in SANAS. If init_temperature and init_tokens are None,
default initial temperature is 10.0, if init_temperature is None and
init_tokens is not None, default initial temperature is 1.0. The detail
configuration about the init_temperature please reference Note. Default: None.
reduce_rate(float): Reduce rate in SANAS. The detail configuration about the reduce_rate please
reference Note. Default: 0.85.
search_steps(int): The steps of searching. Default: 300.
init_tokens(list|None): Initial token. If init_tokens is None, SANAS will random generate initial
tokens. Default: None.
save_checkpoint(string|None): The directory of checkpoint to save, if set to None, not save checkpoint.
Default: 'nas_checkpoint'.
load_checkpoint(string|None): The directory of checkpoint to load, if set to None, not load checkpoint.
Default: None.
is_server(bool): Whether current host is controller server. Default: True.
.. note::
- Why need to set initial temperature and reduce rate:
- SA algorithm preserve a base token(initial token is the first base token, can be set by
yourself or random generate) and base score(initial score is -1), next token will be
generated based on base token. During the search, if the score which is obtained by the
model corresponding to the token is greater than the score which is saved in SA corresponding to
base token, current token saved as base token certainly; if score which is obtained by the model
corresponding to the token is less than the score which is saved in SA correspinding to base token,
current token saved as base token with a certain probability.
- For initial temperature, higher is more unstable, it means that SA has a strong possibility to save
current token as base token if current score is smaller than base score saved in SA.
- For initial temperature, lower is more stable, it means that SA has a small possibility to save
current token as base token if current score is smaller than base score saved in SA.
- For reduce rate, higher means SA algorithm has slower convergence.
- For reduce rate, lower means SA algorithm has faster convergence.
- How to set initial temperature and reduce rate:
- If there is a better initial token, and want to search based on this token, we suggest start search
experiment in the steady state of the SA algorithm, initial temperature can be set to a small value,
such as 1.0, and reduce rate can be set to a large value, such as 0.85. If you want to start search
experiment based on the better token with greedy algorithm, which only saved current token as base
token if current score higher than base score saved in SA algorithm, reduce rate can be set to a
extremely small value, such as 0.85 ** 10.
- If initial token is generated randomly, it means initial token is a worse token, we suggest start
search experiment in the unstable state of the SA algorithm, explore all random tokens as much as
possible, and get a better token. Initial temperature can be set a higher value, such as 1000.0,
and reduce rate can be set to a small value.
"""
def__init__(self,
configs,
server_addr=("",8881),
...
...
@@ -44,21 +104,6 @@ class SANAS(object):
save_checkpoint='nas_checkpoint',
load_checkpoint=None,
is_server=True):
"""
Search a group of ratios used to prune program.
Args:
configs(list<tuple>): A list of search space configuration with format [(key, {input_size, output_size, block_num, block_mask})].
`key` is the name of search space with data type str. `input_size` and `output_size` are
input size and output size of searched sub-network. `block_num` is the number of blocks in searched network, `block_mask` is a list consists by 0 and 1, 0 means normal block, 1 means reduction block.
server_addr(tuple): A tuple of server ip and server port for controller server.
init_temperature(float|None): The init temperature used in simulated annealing search strategy. Default: None.
reduce_rate(float): The decay rate used in simulated annealing search strategy. Default: None.
search_steps(int): The steps of searching. Default: 300.
init_token(list): Init tokens user can set by yourself. Default: None.
save_checkpoint(string|None): The directory of checkpoint to save, if set to None, not save checkpoint. Default: 'nas_checkpoint'.
load_checkpoint(string|None): The directory of checkpoint to load, if set to None, not load checkpoint. Default: None.
is_server(bool): Whether current host is controller server. Default: True.
"""
ifnotis_server:
assertserver_addr[
0]!="","You should set the IP and port of server when is_server is False."
...
...
@@ -149,9 +194,11 @@ class SANAS(object):
deftokens2arch(self,tokens):
"""
Convert tokens to network architectures.
Convert tokens to model architectures.
Args
tokens<list>: A list of token. The length and range based on search space.:
Returns:
list<function>: A list of functions that define networks.
list<function>: A model architecture instance according to tokens.
"""
returnself._search_space.token2arch(tokens)
...
...
@@ -166,9 +213,9 @@ class SANAS(object):
defnext_archs(self):
"""
Get next network architectures.
Get next model architectures.
Returns:
list<function>: A list of functions that define networks.
list<function>: A list of instance of model architecture.
