Abstract
Over the past decade, the development of both adversarial attack methods and defense strategies has accelerated rapidly. Classification accuracy has been predominantly used as the sole metric for assessing model performance. However, when the reported accuracy rates of two models are identical or very similar, it becomes challenging to determine which model is superior. To address this issue and offer more insights into model performance, this study introduces a novel classification performance metric: the confidence gap. This metric is defined as the difference in confidence level between the true label and either the top 1 prediction or the second-best prediction, depending on the accuracy of the image classification. The confidence level, as indicated by its sign, reflects the correctness of the classification and provides more detailed information on the robustness of the classification result. Recognizing that evaluation results may be inconsistent when employing different criteria, we recommend that future research in this field should report the confidence gap alongside accuracy rates.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples, ar**v preprint ar**v:1412.6572 (2014)
Carlini, N., Wagner, D., “Towards evaluating the robustness of neural networks.” In: IEEE Symposium on Security and Privacy (SP), vol. 2017, pp. 39–57. IEEE (2017)
Madry, A., Makelov, A., Schmidt, L., Tsipras, D., Vladu, A.: Towards deep learning models resistant to adversarial attacks. ar**v preprint ar**v:1706.06083 (2017)
Croce, F., Hein, M.: Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. in International Conference on Machine Learning, pp. 2206–2216. PMLR (2020)
Papernot, N., et al.: Technical report on the cleverhans v2.1.0 adversarial examples library. ar**v preprint ar**v:1610.00768 (2018)
Nicolae, M.-I., et al.: Adversarial robustness toolbox v1.2.0. CoRR, vol. 1807.01069 (2018).https://arxiv.org/pdf/1807.01069
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2818–2826 (2016)
Russakovsky, O., et al.: ImageNet large scale visual recognition challenge. Int. J. Comput. Vis. (IJCV) 115(3), 211–252 (2015)
Acknowledgement
This work is partially supported by JSPS international scientific exchanges between Japan and India (Bilateral Program DTS-JSPS) (2022-2024). We thank Prof. Hideki Murahara for his helpful comments, which greatly improved the quality of this manuscript.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Zhang, H., Yao, Z., Sakurai, K. (2023). POSTER: A Fine-Grained Metric for Evaluating the Performance of Adversarial Attacks and Defenses. In: Zhou, J., et al. Applied Cryptography and Network Security Workshops. ACNS 2023. Lecture Notes in Computer Science, vol 13907. Springer, Cham. https://doi.org/10.1007/978-3-031-41181-6_41
Download citation
DOI: https://doi.org/10.1007/978-3-031-41181-6_41
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-41180-9
Online ISBN: 978-3-031-41181-6
eBook Packages: Computer ScienceComputer Science (R0)