Abstract
Aging civil infrastructures are closely monitored by engineers for damage and critical defects. As the manual inspection of such large structures is costly and time-consuming, we are working towards fully automating the visual inspections to support the prioritization of maintenance activities. To that end we combine recent advances in drone technology and deep learning. Unfortunately, annotation costs are incredibly high as our proprietary civil engineering dataset must be annotated by highly trained engineers. Active learning is, therefore, a valuable tool to optimize the trade-off between model performance and annotation costs. Our use-case differs from the classical active learning setting as our dataset suffers from heavy class imbalance and consists of a much larger already labeled data pool than other active learning research. We present a novel method capable of operating in this challenging setting by replacing the traditional active learning acquisition function with an auxiliary binary discriminator. We experimentally show that our novel method outperforms the best-performing traditional active learning method (BALD) by 5% and 38% accuracy on CIFAR-10 and our proprietary dataset respectively.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Matrice 300 RTK - Built Tough. Works Smart. https://www.dji.com/ch/photo
Zenmuse H20 Series - Unleash the Power of One. https://www.dji.com/ch/photo
Bayesian Active Learning (BaaL) (2022). https://github.com/baal-org/baal. Accessed 30 Sept 2019
Aggarwal, U., Popescu, A., Hudelot, C.: Active learning for imbalanced datasets. In: 2020 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1417–1426. IEEE, Snowmass Village (2020). https://doi.org/10.1109/WACV45572.2020.9093475, https://ieeexplore.ieee.org/document/9093475/
Buslaev, A., Iglovikov, V.I., Khvedchenya, E., Parinov, A., Druzhinin, M., Kalinin, A.A.: Albumentations: fast and flexible image augmentations. Information 11(2), 125 (2020). https://doi.org/10.3390/info11020125, https://www.mdpi.com/2078-2489/11/2/125
Chan, K.W., Nirmal, U., Cheaw, W.G.: Progress on drone technology and their applications: a comprehensive review. In: AIP Conference Proceedings, vol. 2030, no. 1, p. 020308 (2018). https://doi.org/10.1063/1.5066949, https://aip.scitation.org/doi/abs/10.1063/1.5066949
Chawla, N.V., Bowyer, K.W., Hall, L.O., Kegelmeyer, W.P.: SMOTE: synthetic minority over-sampling technique. J. Artif. Intell. Res. 16, 321–357 (2002). https://doi.org/10.1613/jair.953, https://www.jair.org/index.php/jair/article/view/10302
Davtalab, O., Kazemian, A., Yuan, X., Khoshnevis, B.: Automated inspection in robotic additive manufacturing using deep learning for layer deformation detection. J. Intell. Manuf. 33(3), 771–784 (2022). https://doi.org/10.1007/s10845-020-01684-w
Feng, C., Liu, M.Y., Kao, C.C., Lee, T.Y.: Deep active learning for civil infrastructure defect detection and classification. In: Computing in Civil Engineering 2017, pp. 298–306. American Society of Civil Engineers, Seattle (2017). https://doi.org/10.1061/9780784480823.036, http://ascelibrary.org/doi/10.1061/9780784480823.036
Freeman, L.C.: Elementary Applied Statistics: For Students in Behavioral Science. Wiley (1965). Google-Books-ID: r4VRAAAAMAAJ
Gal, Y., Ghahramani, Z.: Dropout as a Bayesian approximation: representing model uncertainty in deep learning (2016). ar**v:1506.02142 [cs, stat]
Gawlikowski, J., et al.: A survey of uncertainty in deep neural networks (2022). ar**v:2107.03342 [cs, stat]
Hanneke, S.: Theory of disagreement-based active learning. Found. Trends® Mach. Learn. 7(2–3), 131–309 (2014). https://doi.org/10.1561/2200000037, https://www.nowpublishers.com/article/Details/MAL-037
He, H., Bai, Y., Garcia, E.A., Li, S.: ADASYN: adaptive synthetic sampling approach for imbalanced learning. In: 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence), pp. 1322–1328 (2008). https://doi.org/10.1109/IJCNN.2008.4633969. iSSN 2161-4407
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition (2015). https://doi.org/10.48550/ar**v.1512.03385, ar**v:1512.03385 [cs]
Hernandez, J., Carrasco-Ochoa, J.A., Martínez-Trinidad, J.F.: An empirical study of oversampling and undersampling for instance selection methods on imbalance datasets. In: Ruiz-Shulcloper, J., Sanniti di Baja, G. (eds.) CIARP 2013. LNCS, vol. 8258, pp. 262–269. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-41822-8_33
Houlsby, N., Huszár, F., Ghahramani, Z., Lengyel, M.: Bayesian active learning for classification and preference learning (2011). https://doi.org/10.48550/ar**v.1112.5745, ar**v:1112.5745 [cs, stat]
Intelligence, I.: Drone technology uses and applications for commercial, industrial and military drones in 2021 and the future. https://www.businessinsider.com/drone-technology-uses-applications
Kaur, H., Pannu, H.S., Malhi, A.K.: A systematic review on imbalanced data challenges in machine learning: applications and solutions. ACM Comput. Surv. 52(4), 1–36 (2020). https://doi.org/10.1145/3343440, https://dl.acm.org/doi/10.1145/3343440
Kirsch, A., van Amersfoort, J., Gal, Y.: BatchBALD: efficient and diverse batch acquisition for deep bayesian active learning (2019). ar**v:1906.08158 [cs, stat]
Krizhevsky, A.: Learning multiple layers of features from tiny images, p. 60, 8 April 2009
Kwolek, B., et al.: Breast cancer classification on histopathological images affected by data imbalance using active learning and deep convolutional neural network. In: Tetko, I.V., Kůrková, V., Karpov, P., Theis, F. (eds.) ICANN 2019. LNCS, vol. 11731, pp. 299–312. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-30493-5_31
Lakshminarayanan, B., Pritzel, A., Blundell, C.: Simple and scalable predictive uncertainty estimation using deep ensembles (2017). https://doi.org/10.48550/ar**v.1612.01474, ar**v:1612.01474 [cs, stat]
Li, W., et al.: PathAL: an active learning framework for histopathology image analysis. IEEE Trans. Med. Imaging 41(5), 1176–1187 (2022). https://doi.org/10.1109/TMI.2021.3135002
Liu, Z., Miao, Z., Zhan, X., Wang, J., Gong, B., Yu, S.X.: Large-scale long-tailed recognition in an open world, pp. 2537–2546 (2019). https://openaccess.thecvf.com/content_CVPR_2019/html/Liu_Large-Scale_Long-Tailed_Recognition_in_an_Open_World_CVPR_2019_paper.html
Loshchilov, I., Hutter, F.: Decoupled weight decay regularization (2019). https://doi.org/10.48550/ar**v.1711.05101, ar**v:1711.05101 [cs, math]
Mohammed, R., Rawashdeh, J., Abdullah, M.: Machine learning with oversampling and undersampling techniques: overview study and experimental results. In: 2020 11th International Conference on Information and Communication Systems (ICICS), pp. 243–248 (2020). https://doi.org/10.1109/ICICS49469.2020.239556. iSSN 2573-3346
Ren, R., Hung, T., Tan, K.C.: A generic deep-learning-based approach for automated surface inspection. IEEE Trans. Cybern. 48(3), 929–940 (2018). https://doi.org/10.1109/TCYB.2017.2668395
Richards, J.W., et al.: Active learning to overcome sample selection bias: application to photometric variable star classification. Astrophys. J. 744(2), 192 (2011). https://doi.org/10.1088/0004-637X/744/2/192
Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-CAM: visual explanations from deep networks via gradient-based localization. Int. J. Comput. Vision 128(2), 336–359 (2020). https://doi.org/10.1007/s11263-019-01228-7, ar**v:1610.02391 [cs]
Settles, B.: Active Learning Literature Survey. Technical Report, University of Wisconsin-Madison Department of Computer Sciences (2009), https://minds.wisconsin.edu/handle/1793/60660. Accessed 15–17 Mar 2012
Shannon, C.E.: A mathematical theory of communication p. 55 (1948)
Shi, X., Dou, Q., Xue, C., Qin, J., Chen, H., Heng, P.-A.: An active learning approach for reducing annotation cost in skin lesion analysis. In: Suk, H.-I., Liu, M., Yan, P., Lian, C. (eds.) MLMI 2019. LNCS, vol. 11861, pp. 628–636. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32692-0_72
Wang, X., Liu, B., Cao, S., **g, L., Yu, J.: Important sampling based active learning for imbalance classification. Sci. China Inf. Sci. 63(8), 182104 (2020). https://doi.org/10.1007/s11432-019-2771-0
Wightman, R., et al.: Comar: rwightman/pytorch-image-models: v0.6.5 Release (2022). https://doi.org/10.5281/ZENODO.4414861, https://zenodo.org/record/4414861
Yin, X., Chen, Y., Bouferguene, A., Zaman, H., Al-Hussein, M., Kurach, L.: A deep learning-based framework for an automated defect detection system for sewer pipes. Autom. Constr. 109, 102967 (2020). https://doi.org/10.1016/j.autcon.2019.102967, https://www.sciencedirect.com/science/article/pii/S0926580519307411
Yun, S., Han, D., Oh, S.J., Chun, S., Choe, J., Yoo, Y.: CutMix: regularization strategy to train strong classifiers with localizable features (2019). https://arxiv.org/abs/1905.04899v2
Zhang, H., Cisse, M., Dauphin, Y.N., Lopez-Paz, D.: Mixup: beyond empirical risk minimization. ar**v:1710.09412 [cs, stat] (2018)
Zhang, Y., Kang, B., Hooi, B., Yan, S., Feng, J.: Deep long-tailed learning: a survey (2021). ar**v:2110.04596 [cs]
Acknowledgement
This work would not have been possible without Finn Bormlund and Svend Gjerding from Sund &Bælt. We would like to thank them for their collaboration, specifically for the collection of image data, for their expert annotations, and their tireless help with the annotation guidelines for the civil engineering dataset.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Frick, T., Antognini, D., Rigotti, M., Giurgiu, I., Grewe, B., Malossi, C. (2023). Active Learning for Imbalanced Civil Infrastructure Data. In: Karlinsky, L., Michaeli, T., Nishino, K. (eds) Computer Vision – ECCV 2022 Workshops. ECCV 2022. Lecture Notes in Computer Science, vol 13807. Springer, Cham. https://doi.org/10.1007/978-3-031-25082-8_19
Download citation
DOI: https://doi.org/10.1007/978-3-031-25082-8_19
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-25081-1
Online ISBN: 978-3-031-25082-8
eBook Packages: Computer ScienceComputer Science (R0)