Abstract
We examine the problem of performance evaluation for deep neural networks. We develop a software, which, unlike the existing ones, is focused on evaluating the performance of deep models’ inference on CPUs, integrated graphics and embedded devices. The implementation is open source and free available on GitHub: https://github.com/itlab-vision/openvino-dl-benchmark. The software is verified using the example of the well-known classification model ResNet-152 and the Inference Engine component of the OpenVINO toolkit which is distributed by Intel. The primarily advantage of the OpenVINO toolkit is the absence of restrictions on the choice of a library for model training, since the toolkit contains an utility for converting models into its own intermediate format. We analyze the performance of ResNet-152 in synchronous and asynchronous inference modes on the Intel CPUs and Intel Processor Graphics. We provide recommendations on the selection of the optimal execution parameters. Inference performance results for more than 20 well-known deep models on the available hardware are posted on the project web page: http://hpc-education.unn.ru/dli.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Howard A.G., et al.: MobileNets: efficient convolutional neural networks for mobile vision applications. In: NIPS 2016 Proceedings of the 30th International Conference on Neural Information Processing Systems, pp. 379–387 (2017)
Redmon J., Farhadi A.: YOLOv3: An Incremental Improvement (2018). https://arxiv.org/abs/1804.02767
Chen, L.-C., et al.: DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE Trans. Pattern Anal. Mach. Intell. 40(2018), 834–848 (2018)
George, D., Huerta, E.A.: Deep learning for real-time gravitational wave detection and parameter estimation: results with advanced LIGO data. Phys. Lett. B 778(2018), 64–70 (2017)
Mehta, P., et al.: A high-bias, low-variance introduction to Machine Learning for physicists (2018). https://arxiv.org/abs/1803.08823
Yoon, S.: lncRNAnet: long non-coding RNA identification using deep learning. Bioinformatics 34(22), 3889–3897 (2018)
Yu, T.: A graph-embedded deep feedforward network for disease outcome classification and feature selection using gene expression data. Bioinformatics 34(21), 3727–3737 (2018)
Nemirovsky, D., et al.: A general guide to applying machine learning to computer architecture. Supercomput. Front. Innov. 5(1), 95–115 (2018)
Model Zoo: Discover open source deep learning code and pretrained models. https://modelzoo.co
Caffe Model Zoo. https://github.com/BVLC/caffe/wiki/Model-Zoo
Models and examples built with TensorFlow. https://github.com/tensorflow/models
The models subpackage TORCHVISION.MODELS. https://pytorch.org/docs/stable/torchvision/models.html
Keras Applications: Available models. https://keras.io/applications
Cognitive Toolkit: Model gallery. https://www.microsoft.com/en-us/cognitive-toolkit/features/model-gallery
OpenVINO Toolkit - Open Model Zoo repository. https://github.com/opencv/open_model_zoo
Deep Learning Workstation Solutions: Unlock high-performance AI supercomputing from your desktop. https://www.nvidia.com/en-us/deep-learning-ai/solutions/workstation
Mellanox Enables the Most Efficient Machine Learning Platforms. Allowing AI to Perform Critical & Real Time Decisions to Enhance Competitive Advantages. http://www.mellanox.com/solutions/machine-learning
Intel AI: Hardware. https://ai.intel.com/hardware
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016). https://arxiv.org/abs/1512.03385
Intel Distribution of OpenVINO toolkit. https://software.intel.com/en-us/openvino-toolkit
Deep learning benchmark based on Intel Deep Learning Deployment Toolkit. https://github.com/itlab-vision/openvino-dl-benchmark
DeepBench: Benchmarking Deep Learning operations on different hardware. https://github.com/baidu-research/DeepBench
Deep Learning Benchmark for comparing the performance of DL frameworks, GPUs, and single vs half precision. https://github.com/u39kun/deep-learning-benchmark
DAWNBench: An End-to-End Deep Learning Benchmark and Competition. https://dawn.cs.stanford.edu/benchmark, https://cs.stanford.edu/~deepakn/assets/papers/dawnbench-sosp17.pdf
Deep Learning Benchmarking Suite (DLBS). https://hewlettpackard.github.io/dlcookbook-dlbs/. https://github.com/HewlettPackard/dlcookbook-dlbs
Deep500: A Deep Learning Meta-Framework and HPC Benchmarking Library. https://github.com/deep500/deep500. https://arxiv.org/pdf/1901.10183.pdf
Caffe2: A New Lightweight, Modular, and Scalable Deep Learning Framework. https://caffe2.ai
Apache MXNet (Incubating): A flexible and efficient library for deep learning. mxnet.incubator.apache.org
ImageNET. http://www.image-net.org
ImageNet Large Scale Visual Recognition Competition (ILSVRC). http://www.image-net.org/challenges/LSVRC
DLI: Deep Learning Inference Benchmark (in English). http://hpc-education.unn.ru/dli-ru (In Russian). http://hpc-education.unn.ru/dli
TensorFlow. https://www.tensorflow.org
Caffe: Deep learning framework. http://caffe.berkeleyvision.org
ONNX: Open Neural Network Exchange Format. https://onnx.ai
The ResNet-152 model (.prototxt and.caffemodel files). https://onedrive.live.com/download?cid=4006CBB8476FF777&resid=4006CBB8476FF777%2117893&authkey=AAFW2-FVoxeVRck. https://onedrive.live.com/download?cid=4006CBB8476FF777&resid=4006CBB8476FF777%2117897&authkey=AAFW2-FVoxeVRck
Acknowledgements
The research was supported by the Intel Corporation. The authors thank company’s employees for their help and attention to the research.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Kustikova, V., Vasiliev, E., Khvatov, A., Kumbrasiev, P., Rybkin, R., Kogteva, N. (2019). DLI: Deep Learning Inference Benchmark. In: Voevodin, V., Sobolev, S. (eds) Supercomputing. RuSCDays 2019. Communications in Computer and Information Science, vol 1129. Springer, Cham. https://doi.org/10.1007/978-3-030-36592-9_44
Download citation
DOI: https://doi.org/10.1007/978-3-030-36592-9_44
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-36591-2
Online ISBN: 978-3-030-36592-9
eBook Packages: Computer ScienceComputer Science (R0)