基于递归结构的神经网络架构搜索算法

doi:10.3969/j.issn.1000-5641.2022.04.004

摘要/Abstract

摘要：

神经网络架构搜索算法旨在通过计算机的启发式搜索代替人工搜索, 在巨大的神经网络结构空间中寻找更为高效的神经网络结构. 许多研究通过引入各种对搜索空间的约束来解决早期神经网络结构搜索效率低下、耗时长的问题. 然而, 对于搜索空间的约束虽然能够提升并稳定所搜索到的模型的性能, 但同时也导致了很多潜在的高性能模型结构无法被搜索到. 构建了一种更为关注神经网络宏观结构的递归型搜索空间, 并提出通过分步渐进搜索方案探索这个搜索空间的神经网络架构搜索算法. 实验表明, 该算法能在复杂的搜索空间中高效完成神经网络架构搜索任务, 但与最新的基于受约束搜索空间的神经网络架构搜索算法相比仍稍有差距.

关键词: 神经网络架构搜索, 图像分类, 卷积神经网络

Abstract:

Neural architecture search algorithms aim to find more efficient neural network structures in a huge neural network structure space using computer heuristic search instead of manual search. Previous studies have addressed the problem of inefficient and time-consuming search for early neural network structures by introducing various constraints on the search space. While constraints on the search space can improve and stabilize the performance of the model, they ignore potentially efficient model structures. Hence, in this study, we constructed a recursive model search space that focuses more on the macroscopic structure of neural networks. We proposed a neural architecture search algorithm that explores this search space through a step-by-step incremental search approach. Experiments showed that the algorithm can efficiently perform neural architecture search tasks in complex search spaces, but still fell slightly short of the latest constrained search space-based neural architecture search algorithms.

Key words: neural architecture search, image classification, convolutional neural network

中图分类号:

TP391.4

李继洲, 林欣. 基于递归结构的神经网络架构搜索算法[J]. 华东师范大学学报（自然科学版）, 2022, 2022(4): 31-42.

Jizhou LI, Xin LIN. Neural architecture search algorithms based on a recursive structure[J]. Journal of East China Normal University(Natural Science), 2022, 2022(4): 31-42.

图/表 6

图1

图2

图3

图4

表1

表2

参考文献 23

1	ZOPH B, LE Q V. Neural architecture search with reinforcement learning [EB/OL]. (2016-11-05)[2020-11-10]. https://openreview.net/forum?id=r1Ue8Hcxg.
2	ZOPH B, VASUDEVAN V, SHLENS J, et al. Learning transferable architectures for scalable image recognition [C]// 2018 IEEE Conference on Computer Vision and Pattern Recognition. 2018: 8697-8710.
3	BENDER G, KINDERMANS P J, ZOPH B, et al. Understanding and simplifying one-shot architecture search [C]// Proceedings of the 35th International Conference on Machine Learning. 2018: 550-559.
4	PHAM H, GUAN M Y, ZOPH B, et al. Efficient neural architecture search via parameter sharing [C]// Proceedings of the 35th International Conference on Machine Learning. 2018: 4092-4101.
5	YANG A, ESPERANÇA P M, CARLUCCI F M. NAS evaluation is frustratingly hard [EB/OL]. (2020-02-13)[2020-10-14]. https://arxiv.org/pdf/1912.12522v3.pdf.
6	GUO Z C, ZHANG X Y, MU H Y, et al. Single path one-shot neural architecture search with uniform sampling [C]// Proceedings of the 16th European Conference on Computer Vision. 2020: 544-560.
7	YU K C, SCIUTO C, JAGGI M, et al. Evaluating the search phase of neural architecture search [EB/OL]. (2019-11-22)[2020-11-03]. https://arxiv.org/pdf/1902.08142.pdf.
8	LIU H X, SIMONYAN K, VINYALS O, et al. Hierarchical representations for efficient architecture search [EB/OL]. (2018-02-22)[2020-09-15]. https://arxiv.org/pdf/1711.00436.pdf.
9	ZHONG Z, YAN J J, WU W, et al. Practical block-wise neural network architecture generation [C]// 2018 IEEE Conference on Computer Vision and Pattern Recognition. 2018: 2423-2432.
10	CAI H, CHEN T Y, ZHANG W N, et al. Efficient architecture search by network transformation [C]// Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence. 2018: 2787-2794.
11	REAL E, AGGARWAL A, HUANG Y P, et al. Regularized evolution for image classifier architecture search [C]// The Thirty-Third AAAI Conference on Artificial Intelligence. 2019: 4780-4789.
12	LIANG J, MEYERSON E, MIIKKULAINEN R. Evolutionary architecture search for deep multitask networks [C]// Proceedings of the Genetic and Evolutionary Computation Conference. 2018: 466-473.
13	DONG J D, CHENG A C, JUAN D C, et al. DPP-Net: Device-aware progressive search for pareto-optimal neural architectures [C]// Computer Vision–ECCV 2018. 2018: 540-555.
14	CHU X X, ZHANG B, XU R J, et al. FairNAS: Rethinking evaluation fairness of weight sharing neural architecture search [EB/OL]. (2020-03-10)[2020-09-15]. https://arxiv.org/pdf/1907.01845v4.pdf.
15	LUO R Q, TIAN F, QIN T, et al. Neural architecture optimization [C]// Proceedings of the 32nd Conference on Neural Information Processing Systems. 2018: 7827-7838.
16	YOU S, HUANG T, YANG M M, et al. GreedyNAS: Towards fast one-shot NAS with greedy supernet [C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020: 1996-2005.
17	CAI H, GAN C, WANG T Z, et al. Once-for-all: Train one network and specialize it for efficient deployment [EB/OL]. (2019-09-26)[2020-11-21]. https://openreview.net/forum?id=HylxE1HKwS.
18	LIU H X, SIMONYAN K, YANG Y M. DARTS: Differentiable architecture search [EB/OL]. (2019-04-23)[2020-08-10]. http://export.arxiv.org/pdf/1806.09055.
19	CHU X X, ZHOU T B, ZHANG B, et al. Fair DARTS: Eliminating unfair advantages in differentiable architecture search [C]// Computer Vision-ECCV 2020. 2020: 465-480.
20	XU Y H, XIE L X, ZHANG X P, et al. PC-DARTS: Partial channel connections for memory-efficient architecture search [EB/OL]. (2020-04-07)[2020-08-10]. https://arxiv.org/pdf/1907.05737v4.pdf.
21	CHEN X, XIE L X, WU J, et al. Progressive differentiable architecture search: Bridging the depth gap between search and evaluation [C]// 2019 IEEE/CVF International Conference on Computer Vision. 2019: 1294-1303.
22	HE K, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition [C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition. 2016: 770-778.
23	SANDLER M, HOWARD A, ZHU M L, et al. MobileNetV2: Inverted residuals and linear bottlenecks [C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2018: 4510-4520.

方法	CIFAR-10错误率/%	CIFAR-100错误率/%	参数量	搜索时间/(GPU days)
ResNet-110	4.61		1.70 M
DenseNet-BC	3.46	17.33	25.60 M
DenseNet-Cutout	2.56		26.20 M
NAS^[1]	4.47		7.10 M	22400.00
ENAS + macro search space^[4]	4.23		21.30 M	0.32
one-shot small (F=64)^[3]	3.90		5.00 M	3.30
Hierarchical^[8]	3.63		61.30 M	300.00
NASNet-A + Cutout^[2]	2.65		3.30 M	1800.00
AmoebaNet-A (N=6, F=36)^[11]	3.34	18.93	3.20 M	3150.00
ENAS + micro search space + Cutout^[4]	2.89		4.60 M	0.50
NAONet-WS + Cutout^[15]	2.93		2.50 M	0.30
DARTS (first order) + Cutout^[18]	3.00	17.76	3.30 M	1.50
DARTS (second order) + Cutout^[18]	2.77	17.54	3.30 M	4.00
PC-DARTS^[20]	2.57		3.60 M	0.10
Fair DARTS^[19]	2.54		3.32 M	0.40
P-DARTS^[21]	2.50	16.55	3.40 M	0.30
本文方案 (2 次扩张, 55 通道)	2.97	18.10	3.07 M	0.60
本文方案 (3 次扩张, 40 通道)	2.74	17.64	3.63 M	0.80
本文方案 (DARTS, 3 次扩张, 32 通道)	2.28	15.31	3.50 M	1.40

方案	准确率/%
在完整空间中随机采样 (10 次)	94.36 (±0.30)
第一步使用演化搜索, 第二步使用随机采样 (10 次)	94.48 (±0.25)
第一步使用随机采样, 第二步使用演化搜索 (10 次)	94.55 (±0.21)
直接搜索方案 (3 次)	94.66 (±0.11)
分步搜索方案 (3 次)	94.80 (±0.07)

[1]	刘波, 白晓东, 张更新, 沈俊, 谢继东, 赵来定, 洪涛. 深度学习在认知无线电中的应用研究综述[J]. 华东师范大学学报（自然科学版）, 2021, 2021(1): 36-52.
[2]	贺小娟, 郭新顺. 基于特征优化的广告点击率预测模型研究[J]. 华东师范大学学报（自然科学版）, 2020, 2020(4): 147-155.
[3]	袁培森, 张勇, 李美玲, 顾兴健. 基于深度哈希学习的商标图像检索研究[J]. 华东师范大学学报(自然科学版), 2018, 2018(5): 172-182.
[4]	金丽娇, 傅云斌, 董启文. 基于卷积神经网络的自动问答[J]. 华东师范大学学报(自然科学版), 2017, 2017(5): 66-79.
[5]	熊轶群;吴健平. 面向对象的城市绿地信息提取方法研究[J]. 华东师范大学学报(自然科学版), 2006, 2006(4): 84-90.