Abstract
Firmware analysis methods are crucial for IoT security, yet their reproducibility-the ability to replicate results in subsequent research-has not been thoroughly examined. This study addresses this gap by empirically analyzing the reproducibility of three methods in two key applications of firmware analysis: third-party library identification and binary image base determination. We then evaluate the original studies on each of these methods, using two reproducibility assessment techniques, providing insights into the challenges and opportunities related to reproducibility in firmware analysis. Our findings highlight the current reproducibility status of these methods and offer guidance for improving the reliability of research in this field.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Abt, S., Stampp, R., Baier, H.: Towards reproducible cyber-security research through complex node automation. In: Badra, M., Boukerche, A., Urien, P. (eds.) 7th International Conference on New Technologies, Mobility and Security, NTMS 2015, Paris, France, 27–29 July 2015, pp. 1–5. IEEE (2015). https://doi.org/10.1109/NTMS.2015.7266527
Akiyama, M., Shiraishi, S., Fukumoto, A., Yoshimoto, R., Shioji, E., Yamauchi, T.: Seeing is not always believing: insights on iot manufacturing from firmware composition analysis and vendor survey. Comput. Secur. 133, 103389 (2023). https://doi.org/10.1016/J.COSE.2023.103389
Cheng, Y., Chen, W., Fan, W., Huang, W., Yu, G., Liu, W.: Iotfuzzbench: a pragmatic benchmarking framework for evaluating iot black-box protocol fuzzers. Electronics 12(14), 3010 (2023)
González-Barahona, J.M., Robles, G.: On the reproducibility of empirical software engineering studies based on data retrieved from development repositories. Empir. Softw. Eng. 17(1–2), 75–89 (2012). https://doi.org/10.1007/S10664-011-9181-9
Helmke, R., Padilla, E., Aschenbruck, N.: Corpus christi: establishing replicability when sharing the bread is not allowed. ar**v preprint ar**v:2404.11977 (2024)
Hernandez, G., et al.: Firmwire: transparent dynamic analysis for cellular baseband firmware. In: 29th Annual Network and Distributed System Security Symposium, NDSS 2022, San Diego, California, USA, 24–28 April 2022. The Internet Society (2022). https://www.ndss-symposium.org/ndss-paper/auto-draft-200/
Juristo, N., Vegas, S.: Using differences among replications of software engineering experiments to gain knowledge. In: 2009 3Rd International Symposium on Empirical Software Engineering and Measurement, pp. 356–366. IEEE (2009)
Muench, M.: Dynamic binary firmware analysis: challenges & solutions. (Analyse dynamique de micrologiciels binaires: défis et solutions). Ph.D. thesis, Sorbonne University, France (2019). https://tel.archives-ouvertes.fr/tel-03143960
Neto, F.G.D.O., Torkar, R., Machado, P.D.: An initiative to improve reproducibility and empirical evaluation of software testing techniques. In: 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, vol. 2, pp. 575–578. IEEE (2015)
Olivier, P., Ngo, X., Francillon, A.: BEERR: bench of embedded system experiments for reproducible research. In: IEEE European Symposium on Security and Privacy, EuroS &P 2022 - Workshops, Genoa, Italy, 6–10 June 2022, pp. 332–339. IEEE (2022). https://doi.org/10.1109/EUROSPW55150.2022.00040
Peffers, K., et al.: The design science research process: a model for producing and presenting information systems research. In: 1st International Conference, pp. 83–106 (2006)
Raghupathi, W., Raghupathi, V., Ren, J.: Reproducibility in computing research: an empirical study. IEEE Access 10, 29207–29223 (2022)
Rahman, M.M., Khomh, F., Castelluccio, M.: Works for me! cannot reproduce-a large scale empirical study of non-reproducible bugs. Empir. Softw. Eng. 27(5), 111 (2022)
Rodríguez-Pérez, G., Robles, G., González-Barahona, J.M.: Reproducibility and credibility in empirical software engineering: a case study based on a systematic literature review of the use of the SZZ algorithm. Inf. Softw. Technol. 99, 164–176 (2018)
Yang, C., Xu, Z., Chen, H., Liu, Y., Gong, X., Liu, B.: Modx: Binary level partially imported third-party library detection via program modularization and semantic matching. In: 44th IEEE/ACM 44th International Conference on Software Engineering, ICSE 2022, Pittsburgh, PA, USA, 25–27 May 2022, pp. 1393–1405. ACM (2022). https://doi.org/10.1145/3510003.3510627
Zhu, R., Tan, Y., Zhang, Q., Wu, F., Zheng, J., Xue, Y.: Determining image base of firmware files for ARM devices. IEICE Trans. Inf. Syst. 99-D(2), 351–359 (2016). https://doi.org/10.1587/TRANSINF.2015EDP7217
Zhu, R., Tan, Y.A., Zhang, Q., Li, Y., Zheng, J.: Determining image base of firmware for arm devices by matching literal pools. Digital Invest. 16, 19–28 (2016)
Zhu, R., Zhang, B., Mao, J., Zhang, Q., Tan, Y.A.: A methodology for determining the image base of arm-based industrial control system firmware. Int. J. Crit. Infrastruct. Prot. 16, 26–35 (2017)
Zhu, R., Zhang, B., Tan, Y.A., Wan, Y., Wang, J.: Determining the image base of arm firmware by matching function addresses. Wirel. Commun. Mobile Comput. 2021, 1–10 (2021)
Zhu, R., Zhang, B., Tan, Y.A., Wang, J., Wan, Y.: Determining the image base of smart device firmware for security analysis. Wirel. Commun. Mobile Comput. 2020, 1–12 (2020)
Acknowledgment
Narges Yousefnezhad acknowledges the support of Jenny and the Antti Wihuri Foundation through the PoDoCo program (www.podoco.fi), grant number 141222. (Part of) This work was supported by the European Commission under the Horizon Europe Programme, as part of the project LAZARUS (https://lazarus-he.eu/) (Grant Agreement no. 101070303). The content of this article does not reflect the official opinion of the European Union. Responsibility for the information and views expressed therein lies entirely with the authors. (Part of this work was) Funded by the European Union (Grant Agreement Nr. 101120962, RESCALE Project). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the Health and Digital Executive Agency. Neither the European Union nor the granting authority can be held responsible for them.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Yousefnezhad, N., Costin, A. (2024). Reproducibility of Firmware Analysis: An Empirical Study. In: Shishkov, B. (eds) Business Modeling and Software Design. BMSD 2024. Lecture Notes in Business Information Processing, vol 523. Springer, Cham. https://doi.org/10.1007/978-3-031-64073-5_13
Download citation
DOI: https://doi.org/10.1007/978-3-031-64073-5_13
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-64072-8
Online ISBN: 978-3-031-64073-5
eBook Packages: Computer ScienceComputer Science (R0)