Nerual Architecture Search for Image Classification¶

This tutorial shows how to use API about SANAS in PaddleSlim. We start experiment based on MobileNetV2 as example. The tutorial contains follow section.

necessary imports
initial SANAS instance
define function about building program
define function about input data
define function about training
define funciton about evaluation
start search 7.1 fetch model architecture 7.2 build program 7.3 define input data 7.4 train model 7.5 evaluate model 7.6 reture score
full example

The following chapter describes each steps in order.

1. import dependency¶

Please make sure that you haved installed Paddle correctly, then do the necessary imports.

import paddle
import paddle.fluid as fluid
import paddleslim as slim
import numpy as np

2. initial SANAS instance¶

sanas = slim.nas.SANAS(configs=[('MobileNetV2Space')], server_addr=("", 8337), save_checkpoint=None)

3. define function about building program¶

Build program about training and evaluation according to the model architecture.

def build_program(archs):
    train_program = fluid.Program()
    startup_program = fluid.Program()
    with fluid.program_guard(train_program, startup_program):
        data = fluid.data(name='data', shape=[None, 3, 32, 32], dtype='float32')
        label = fluid.data(name='label', shape=[None, 1], dtype='int64')
        output = archs(data)
        output = fluid.layers.fc(input=output, size=10)

        softmax_out = fluid.layers.softmax(input=output, use_cudnn=False)
        cost = fluid.layers.cross_entropy(input=softmax_out, label=label)
        avg_cost = fluid.layers.mean(cost)
        acc_top1 = fluid.layers.accuracy(input=softmax_out, label=label, k=1)
        acc_top5 = fluid.layers.accuracy(input=softmax_out, label=label, k=5)
        test_program = fluid.default_main_program().clone(for_test=True)

        optimizer = fluid.optimizer.Adam(learning_rate=0.1)
        optimizer.minimize(avg_cost)

        place = fluid.CPUPlace()
        exe = fluid.Executor(place)
        exe.run(startup_program)
    return exe, train_program, test_program, (data, label), avg_cost, acc_top1, acc_top5

4. define function about input data¶

The dataset we used is cifar10, and paddle.dataset.cifar in Paddle including the download and pre-read about cifar.

def input_data(inputs):
    train_reader = paddle.batch(paddle.reader.shuffle(paddle.dataset.cifar.train10(cycle=False), buf_size=1024),batch_size=256)
    train_feeder = fluid.DataFeeder(inputs, fluid.CPUPlace())
    eval_reader = paddle.batch(paddle.dataset.cifar.test10(cycle=False), batch_size=256)
    eval_feeder = fluid.DataFeeder(inputs, fluid.CPUPlace())
    return train_reader, train_feeder, eval_reader, eval_feeder

5. define function about training¶

Start training.

def start_train(program, data_reader, data_feeder):
    outputs = [avg_cost.name, acc_top1.name, acc_top5.name]
    for data in data_reader():
        batch_reward = exe.run(program, feed=data_feeder.feed(data), fetch_list = outputs)
        print("TRAIN: loss: {}, acc1: {}, acc5:{}".format(batch_reward[0], batch_reward[1], batch_reward[2]))

6. define funciton about evaluation¶

Start evaluating.

def start_eval(program, data_reader, data_feeder):
    reward = []
    outputs = [avg_cost.name, acc_top1.name, acc_top5.name]
    for data in data_reader():
        batch_reward = exe.run(program, feed=data_feeder.feed(data), fetch_list = outputs)
        reward_avg = np.mean(np.array(batch_reward), axis=1)
        reward.append(reward_avg)
        print("TEST: loss: {}, acc1: {}, acc5:{}".format(batch_reward[0], batch_reward[1], batch_reward[2]))
    finally_reward = np.mean(np.array(reward), axis=0)
    print("FINAL TEST: avg_cost: {}, acc1: {}, acc5: {}".format(finally_reward[0], finally_reward[1], finally_reward[2]))
    return finally_reward

7. start search¶

The following steps describes how to get current model architecture and what need to do after get the model architecture. If you want to start a full example directly, please jump to Step 9.

7.1 fetch model architecture¶

Get Next model architecture by next_archs().

archs = sanas.next_archs()[0]

7.2 build program¶

Get program according to the function in Step3 and model architecture from Step 7.1.

exe, train_program, eval_program, inputs, avg_cost, acc_top1, acc_top5 = build_program(archs)

7.3 define input data¶

train_reader, train_feeder, eval_reader, eval_feeder = input_data(inputs)

7.4 train model¶

Start training according to train program and data.

start_train(train_program, train_reader, train_feeder)

7.5 evaluate model¶

Start evaluation according to evaluation program and data.

finally_reward = start_eval(eval_program, eval_reader, eval_feeder)

7.6 reture score¶

sanas.reward(float(finally_reward[1]))

8. full example¶

The following is a full example about neural architecture search, it uses FLOPs as constraint and includes 3 steps, it means train 3 model architectures which is satisfied constraint, and train 7 epoch for each model architecture.

for step in range(3):
    archs = sanas.next_archs()[0]
    exe, train_program, eval_progarm, inputs, avg_cost, acc_top1, acc_top5 = build_program(archs)
    train_reader, train_feeder, eval_reader, eval_feeder = input_data(inputs)

    current_flops = slim.analysis.flops(train_program)
    if current_flops > 321208544:
        continue

    for epoch in range(7):
        start_train(train_program, train_reader, train_feeder)

    finally_reward = start_eval(eval_program, eval_reader, eval_feeder)

    sanas.reward(float(finally_reward[1]))