Abstract
Edge computing aims to enable edge devices, such as IoT devices, to process data locally instead of relying on the cloud. However, deep learning techniques like computer vision and natural language processing can be computationally expensive and memory-intensive. Creating manual architectures specialized for each device is infeasible due to their varying memory and computational constraints. To address these concerns, we automate the construction of task-specific deep learning architectures optimized for device constraints through Neural Architecture Search (NAS). We present DCA-NAS, a principled method of fast neural network architecture search that incorporates edge-device constraints such as model size and floating-point operations. It incorporates weight sharing and channel bottleneck techniques to speed up the search time. Based on our experiments, we see that DCA-NAS outperforms manual architectures for similar sized models and is comparable to popular mobile architectures on various image classification datasets like CIFAR-10, CIFAR-100, and Imagenet-1k. Experiments with search spaces—DARTS and NAS-Bench-201 show the generalization capabilities of DCA-NAS. On further evaluating our approach on Hardware-NAS-Bench, device-specific architectures with low inference latency and state-of-the-art performance were discovered.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Abai, Z., Rajmalwar, N.: DenseNet models for tiny ImageNet classification (2020)
Cai, H., Gan, C., Wang, T., Zhang, Z., Han, S.: Once-for-all: train one network and specialize it for efficient deployment (2020). ar**v:1908.09791 [cs, stat]
Cai, H., Zhu, L., Han, S.: ProxylessNAS: direct neural architecture search on target task and hardware (2019)
Chen, X., Hsieh, C.J.: Stabilizing differentiable architecture search via perturbation-based regularization. In: International Conference on Machine Learning, pp. 1554–1565. PMLR (2020)
Chen, X., Wang, R., Cheng, M., Tang, X., Hsieh, C.J.: DRNAS: Dirichlet neural architecture search. ar**v preprint ar**v:2006.10355 (2020)
Chu, G., et al.: Discovering multi-hardware mobile models via architecture search. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3022–3031 (2021)
Ding, Y., et al.: NAP: neural architecture search with pruning. Neurocomputing 477, 85–95 (2022)
Dong, X., Yang, Y.: Searching for a robust neural architecture in four GPU hours. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1761–1770 (2019)
Han, D., Kim, J., Kim, J.: Deep pyramidal residual networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5927–5935 (2017)
He, C., Ye, H., Shen, L., Zhang, T.: MileNAS: efficient neural architecture search via mixed-level reformulation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11993–12002 (2020)
He, Y., Zhang, X., Sun, J.: Channel pruning for accelerating very deep neural networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1389–1397 (2017)
Iandola, F.N., Han, S., Moskewicz, M.W., Ashraf, K., Dally, W.J., Keutzer, K.: SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and \(<\)0.5mb model size (2016)
Jiang, Q., Zhang, X., Chen, D., Do, M.N., Yeh, R.A.: EH-DNAS: end-to-end hardware-aware differentiable neural architecture search. ar**v:2111.12299 [cs] (2021)
**, X., et al.: RC-DARTS: resource constrained differentiable architecture search. ar**v preprint ar**v:1912.12814 (2019)
Kim, J.H., Choo, W., Song, H.O.: Puzzle mix: exploiting saliency and local statistics for optimal mixup (2020)
Li, G., Qian, G., Delgadillo, I.C., Müller, M., Thabet, A., Ghanem, B.: SGAS: sequential greedy architecture search (2020)
Lin, J., Chen, W.M., Lin, Y., Gan, C., Han, S., et al.: MCUNet: tiny deep learning on IoT devices. Adv. Neural. Inf. Process. Syst. 33, 11711–11722 (2020)
Liu, H., Simonyan, K., Yang, Y.: DARTS: differentiable architecture search. ar**v preprint ar**v:1806.09055 (2018)
Lyu, B., Yuan, H., Lu, L., Zhang, Y.: Resource-constrained neural architecture search on edge devices. IEEE Trans. Netw. Sci. Eng. 9(1), 134–142 (2022). https://doi.org/10.1109/TNSE.2021.3054583
Ma, N., Zhang, X., Zheng, H.T., Sun, J.: ShuffleNet V2: practical guidelines for efficient CNN architecture design. In: Proceedings of the European conference on computer vision (ECCV), pp. 116–131 (2018)
Pham, H., Guan, M.Y., Zoph, B., Le, Q.V., Dean, J.: Efficient neural architecture search via parameter sharing. In: ICML (2018)
Real, E., Aggarwal, A., Huang, Y., Le, Q.V.: Regularized evolution for image classifier architecture search (2019)
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.C.: MobileNetV 2: inverted residuals and linear bottlenecks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4510–4520 (2018)
Srivastava, A., Dutta, O., Gupta, J., Agarwal, S., Prathosh, A.P.: A variational information bottleneck based method to compress sequential networks for human action recognition. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2745–2754 (2021)
Tan, M., et al.: platform-aware neural architecture search for mobile. In: 2019 IEEE CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2815–2823 (2019)
Tan, M., Le, Q.: EfficientNet: rethinking model scaling for convolutional neural networks. In: International Conference on Machine Learning, pp. 6105–6114. PMLR (2019)
Wang, R., Cheng, M., Chen, X., Tang, X., Hsieh, C.J.: Rethinking architecture selection in differentiable NAS. ar**v preprint ar**v:2108.04392 (2021)
Wistuba, M.: Deep learning architecture search by neuro-cell-based evolution with function-preserving mutations. In: Berlingerio, M., Bonchi, F., Gärtner, T., Hurley, N., Ifrim, G. (eds.) ECML PKDD 2018. LNCS (LNAI), vol. 11052, pp. 243–258. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-10928-8_15
Wu, Y., et al.: Compiler-aware neural architecture search for on-mobile real-time super-resolution (2022). ar**v:2207.12577 [cs, eess]
**ao, H., Wang, Z., Zhu, Z., Zhou, J., Lu, J.: Shapley-NAS: discovering operation contribution for neural architecture search (2022). ar**v:2206.09811 [cs]
**e, S., Zheng, H., Liu, C., Lin, L.: SNAS: stochastic neural architecture search. In: International Conference on Learning Representations (2018)
**ong, Y., Mehta, R., Singh, V.: Resource constrained neural network architecture search: will a submodularity assumption help? In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1901–1910 (2019)
Xu, Y., et al.: PC-DARTS: partial channel connections for memory-efficient architecture search. ar**v preprint ar**v:1907.05737 (2019)
Yang, Y., You, S., Li, H., Wang, F., Qian, C., Lin, Z.: Towards improving the consistency, efficiency, and flexibility of differentiable neural architecture search. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6667–6676 (2021)
Zhou, H., Yang, M., Wang, J., Pan, W.: BayesNAS: a Bayesian approach for neural architecture search (2019)
Acknowledgement
We thank the anonymous reviewers; Profs. Surendra Prasad and Brejesh Lall of IIT Delhi; and colleagues at Cadence India for their valuable feedback and inputs. This research is supported by funding from Cadence India; the first author is also supported by a fellowship from the Ministry of Education, India.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Dutta, O., Kanvar, T., Agarwal, S. (2023). Search-Time Efficient Device Constraints-Aware Neural Architecture Search. In: Maji, P., Huang, T., Pal, N.R., Chaudhury, S., De, R.K. (eds) Pattern Recognition and Machine Intelligence. PReMI 2023. Lecture Notes in Computer Science, vol 14301. Springer, Cham. https://doi.org/10.1007/978-3-031-45170-6_5
Download citation
DOI: https://doi.org/10.1007/978-3-031-45170-6_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-45169-0
Online ISBN: 978-3-031-45170-6
eBook Packages: Computer ScienceComputer Science (R0)