Abstract
In this work, we present iDropout, a new method to adjust dropout, from purely randomly drop** inputs to drop** inputs based on a mix based on the relevance of the nodes and some randomness. We use Deep Taylor Decomposition to calculate the respective relevance of the inputs and based on this, we give input nodes with a higher relevance a higher probability to be included than input nodes that seem to have less of an impact. The proposed method does not only seem to increase the performance of a Neural Network, but it also seems to make the network more robust to missing data. We evaluated the approach on artificial data with various settings, e.g. noise in data, number of informative features and on real-world datasets from the UCI Machine Learning Repository.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
References
Alber, M., et al.: iNNvestigate neural networks! ar**v:1808.04260 [cs, stat], August 2018
Ba, J., Frey, B.: Adaptive dropout for training deep neural networks. In: Advances in Neural Information Processing Systems, pp. 3084–3092 (2013)
Bacciu, D., Crecchi, F.: Augmenting recurrent neural networks resilience by dropout. IEEE Trans. Neural Netw. Learn. Syst. 1–7 (2019). https://doi.org/10.1109/TNNLS.2019.2899744. https://ieeexplore.ieee.org/document/8668686/
Bacciu, D., Crecchi, F., Morelli, D.: DropIn: making reservoir computing neural networks robust to missing inputs by dropout. ar**v:1705.02643 [cs, stat], May 2017
Dua, D., Graff, C.: UCI machine learning repository (2019). http://archive.ics.uci.edu/ml
Garca-Laencina, P.J., Sancho-Gmez, J.L., Figueiras-Vidal, A.R., Verleysen, M.: K nearest neighbours with mutual information for simultaneous classification and missing data imputation. Neurocomputing 72(7–9), 1483–1493 (2009). https://doi.org/10.1016/j.neucom.2008.11.026. https://linkinghub.elsevier.com/retrieve/pii/S0925231209000149
Guyon, I.: Design of experiments for the NIPS 2003 variable selection benchmark, p. 30 (2003)
Keshari, R., Singh, R., Vatsa, M.: Guided dropout. ar**v preprint ar**v:1812.03965 (2018)
Li, Y., Parker, L.E.: Nearest neighbor imputation using spatial-temporal correlations in wireless sensor networks. Inf. Fusion 15, 64–79 (2014). https://doi.org/10.1016/j.inffus.2012.08.007. https://linkinghub.elsevier.com/retrieve/pii/S1566253512000711
Liu, Z.G., Pan, Q., Dezert, J., Martin, A.: Adaptive imputation of missing values for incomplete pattern classification. Pattern Recogn. 52, 85–95 (2016). https://doi.org/10.1016/j.patcog.2015.10.001. http://arxiv.org/abs/1602.02617
Mahmood, M.A., Seah, W.K., Welch, I.: Reliability in wireless sensor networks: a survey and challenges ahead. Comput. Netw. 79, 166–187 (2015). https://doi.org/10.1016/j.comnet.2014.12.016. https://linkinghub.elsevier.com/retrieve/pii/S1389128614004708
Montavon, G., Lapuschkin, S., Binder, A., Samek, W., Mller, K.R.: Explaining nonlinear classification decisions with deep Taylor decomposition. Pattern Recogn. 65, 211–222 (2017). https://doi.org/10.1016/j.patcog.2016.11.008. https://linkinghub.elsevier.com/retrieve/pii/S0031320316303582
Montavon, G., Samek, W., Mller, K.R.: Methods for interpreting and understanding deep neural networks. Digit. Signal Process. 73, 1–15 (2018). https://doi.org/10.1016/j.dsp.2017.10.011. https://linkinghub.elsevier.com/retrieve/pii/S1051200417302385
Morris, A.C., Josifovski, L., Bourlard, H., Cooke, M., Green, P.: A neural network for classification with incomplete data: application to robust ASR. In: Proceedings of ICSLP, p. 4 (2000)
Singh, N., Javeed, A., Chhabra, S., Kumar, P.: Missing value imputation with unsupervised kohonen self organizing map. In: Shetty, N.R., Prasad, N.H., Nalini, N. (eds.) Emerging Research in Computing, Information, Communication and Applications, pp. 61–76. Springer, New Delhi (2015). https://doi.org/10.1007/978-81-322-2550-8_7
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overtting. J. Mach. Learn. Res. 15(1), 1929–1958 (2014)
Thirukumaran, S., Sumathi, A.: Missing value imputation techniques depth survey and an imputation algorithm to improve the efficiency of imputation. In: Fourth International Conference on Advanced Computing (ICoAC), pp. 1–5. IEEE, Chennai, December 2012. https://doi.org/10.1109/ICoAC.2012.6416805
Wan, L., Zeiler, M., Zhang, S., LeCun, Y., Fergus, R.: Regularization of neural networks using DropConnect. In: International Conference on Machine Learning, p. 9 (2013)
Wang, S., et al.: Defensive dropout for hardening deep neural networks under adversarial attacks. In: Proceedings of the International Conference on Computer-Aided Design - ICCAD 2018, pp. 1–8 (2018). https://doi.org/10.1145/3240765.3264699. http://arxiv.org/abs/1809.05165
Li, Y., Parker, L.: A spatial-temporal imputation technique for classification with missing data in a wireless sensor network. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 3272–3279. IEEE, Nice, September 2008. https://doi.org/10.1109/IROS.2008.4650774
Acknowledgments
This research was supported by the German Federal Ministry for Economic Affairs and Energy (Grant No. 01MD18011D) and the German Federal Ministry of Transport and Digital Infrastructure (Grant No. 16AVF2139F).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Schreckenberger, C., Bartelt, C., Stuckenschmidt, H. (2019). iDropout: Leveraging Deep Taylor Decomposition for the Robustness of Deep Neural Networks. In: Panetto, H., Debruyne, C., Hepp, M., Lewis, D., Ardagna, C., Meersman, R. (eds) On the Move to Meaningful Internet Systems: OTM 2019 Conferences. OTM 2019. Lecture Notes in Computer Science(), vol 11877. Springer, Cham. https://doi.org/10.1007/978-3-030-33246-4_7
Download citation
DOI: https://doi.org/10.1007/978-3-030-33246-4_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-33245-7
Online ISBN: 978-3-030-33246-4
eBook Packages: Computer ScienceComputer Science (R0)