Relative Neural Architecture Search via Slow-Fast Learning

First author：Tan Hao [PDF]

NAS: Neural Architecture Search 神经架构搜素

automating the design of artificial neural networks

Motivation

To benefit form the merits while overcoming the deficiencies of the differentiable NAS and population-based NAS.

Deficiencies

Differentiable NAS

Search by gradient can be ineffective due to the lack of proper diversity

Population-Based NAS

Search efficiency is poor due to the stochastic crossover/mutation and a large number of performance evaluations.

Method

continuous encoding scheme

Spired by H. Liu, K. Simonyan, and Y. Yang, “DARTS: Differentiable architecture search,” in Proceedings of the International Conference on Learning Representations, 2019.

Cell-based architecture

Two types of cells: the normal cell and the reduction cell(down sampling to reduce the size of the feature)

Encodes the node and the operation separately(represented by a real value interval)

The network is a DAG(Directed acyclic graph)

different with DARTS: 1) no requirement of differentiability 2) directly encode the operations between pairwise nodes into real values

Pros: provide more flexibility and versatility

networks can achieve promising performance and high transferability for different tasks by adjusting the total number of cells in the final architecture

slow-fast learning paradigm

inspire by *Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, Kaiming He*; Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019, pp. 6202-6211 , which proposed use Fast pathway(high frame rate) and Slow pathway(low frame rate) in model for video recognition

in each pair of architecture vectors, conside the one with worse performance as slow-learning and the one with better performance as fast-learning.

architecture vectors are upated by pseudo-gradient mechanism which detemined by slow-learning and fast-learning

At each generation, the population is randomly divided into N/2N/2N/2 pairs, αp,sg\boldsymbol{\alpha}_{p, s}^{g}αp,sg is updated by learning form αp,fg\boldsymbol{\alpha}_{p, f}^{g}αp,fg with:

Δαp,sg=λ1(αp,fg−αp,sg)+λ2Δαp,sg−1\Delta \boldsymbol{\alpha}_{p, s}^{g}=\lambda_{1}\left(\boldsymbol{\alpha}_{p, f}^{g}-\boldsymbol{\alpha}_{p, s}^{g}\right)+\lambda_{2} \Delta \boldsymbol{\alpha}_{p, s}^{g-1}Δαp,sg=λ1(αp,fg−αp,sg)+λ2Δαp,sg−1

Due to pseudo-gradient based mechanism, the RelativeNAS is applicable any other generic continuously encoded search space

noval performance estimation strategy

adopt an operation collection as a weight set to estimate the performances

the weight set is not directly trained but update in an online manner

RelativeNAS is intuitively feasible to use performance estimations to obtain the approximate validation losses of the candidate architectures.

Pros: save substantial computation costs

Result

Pick CIFAR-10 as dataset

It only takes about nine hours with a unique 1080Ti or seven hours with a Tesla V100 to complete the above search procedure.
RelativeNAS + Cutout has low Test Error(2.34%) and middle Params(3.93M) , efficient

Transferability Analyses

Intra-task Transferability: CIFAR-100, ImageNet
Inter-task Transferability: Object Detection, Semantic Segmentation, Keypoint Detection

Conclusion

This work highlight the merits of differentiable NAS and combining population-based NAS, to be more effective and more efficient. Moreover, the proposed slow-fast learning paradigm can be also potentially applicable to other generic learning/optimization tasks.

[RelativeNAS] Relative Neural Architecture Search via Slow-Fast Learning相关推荐

架构搜索文献笔记（10）：《Fast Hardware-Aware Neural Architecture Search》
***论文< Fast Hardware-Aware Neural Architecture Search>-2020-CVPR ***** 论文链接一.摘要及主要贡献 1.1 摘要论 ...
Fast AI ： An Opinionated Introduction to AutoML and Neural Architecture Search
An Opinionated Introduction to AutoML and Neural Architecture Search
神经架构搜索(Neural Architecture Search,NAS)介绍
神经架构搜索Neural Architecture Search,NAS介绍 Introduction Intractable Search Space Non transferable optima ...
Neural Architecture Search: A survey
文章目录 1. Introduction 2. Search Space 2.1搜索空间定义: 2.2 常见的搜索空间举例: 2.2.1 简单链式搜索空间: 2.2.2 复杂多分支搜索空间 2.2.3 ...
【读点论文】FBNetV2:Differentiable Neural Architecture Search for Spatial and Channel D扩大搜索空间，复用featuremap
FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensions Abstract 可微分神经 ...
自学网络结构（一）：Neural Architecture Search With Reinforcement Learning
论文:Neural Architecture Search With Reinforcement Learning 链接:https://arxiv.org/abs/1611.01578 代码链接:h ...
神经网络架构搜索（Neural Architecture Search）杂谈
一.背景机器学习从业者被戏称为"调参工"已经不是一天两天了.我们知道,机器学习算法的效果好坏不仅取决于参数,而且很大程度上取决于各种超参数.有些paper的结果很难重现原因之一就 ...
【读点论文】MnasNet: Platform-Aware Neural Architecture Search for Mobile，用神经网络搜索的方式来设计网络平衡精度与速度
MnasNet: Platform-Aware Neural Architecture Search for Mobile Abstract 为移动设备设计卷积神经网络(CNN)模型具有挑战性,因为移 ...
神经网络架构搜索（Neural Architecture Search, NAS）笔记
目录 (一)背景 (二)NAS流程 2.1 定义搜索空间 2.2 搜索策略 (三)加速 (四)变体及扩展 4.1 扩展到其他任务 4.2 扩展到其他超参数 (一)背景机器学习从业者被戏称为" ...

[RelativeNAS] Relative Neural Architecture Search via Slow-Fast Learning