Abstract
We study the problem of feature selection in general machine learning (ML) context, which is one of the most critical subjects in the field. Although, there exist many feature selection methods, however, these methods face challenges such as scalability, managing high-dimensional data, dealing with correlated features, adapting to variable feature importance, and integrating domain knowledge. To this end, we introduce the “Adaptive Feature Selection with Binary Masking” (AFS-BM) which remedies these problems. AFS-BM achieves this by joint optimization for simultaneous feature selection and model training. In particular, we do the joint optimization and binary masking to continuously adapt the set of features and model parameters during the training process. This approach leads to significant improvements in model accuracy and a reduction in computational requirements. We provide an extensive set of experiments where we compare AFS-BM with the established feature selection methods using well-known datasets from real-life competitions. Our results show that AFS-BM makes significant improvement in terms of accuracy and requires significantly less computational complexity. This is due to AFS-BM’s ability to dynamically adjust to the changing importance of features during the training process, which an important contribution to the field. We openly share our code for the replicability of our results and to facilitate further research.
This is a preview of subscription content, log in via an institution to check access.
Data Availability
The data that support the findings of this study is openly available in UCI Machine Learning Repository at https://archive.ics.uci.edu.
Notes
The rest of the paper is organized as follows: Sect. 2 outlines the current literature of the filter, wrapper, embedded and adaptive feature selection methods. Section 3 presents the mathematical background and problem description with current feature selection methods. Section 4 introduces our novel feature selection structure, detailing the algorithms and their underlying principles. Section 5 showcases our experimental results, comparing our approach with established feature selection techniques. Finally, Sect. 7 offers a summary of our findings and conclusions.
For a given vector \( \varvec{x} \) and a matrix \(\varvec{X}\), their respective transposes are represented as \( \varvec{x}^T \) and \( \varvec{X}^T \). The symbol \( \odot \) denotes the Hadamard product, which signifies an element-wise multiplication operation between matrices. For any vector \( \varvec{x} \), \( x_i \) represents the \( i^{th} \) element. For a matrix \( \varvec{X} \), \( X_{ij} \) indicates the element in the \( i^{th} \) row and \( j^{th} \) column. The operation \( \sum (\cdot ) \) calculates the sum of the elements of a vector or matrix. The L1 norm of a vector \( \varvec{x} \) is defined as \(||\varvec{x}||_1 = \sum _{i} |x_i|,\) where the summation runs over all elements of the vector. The number of elements in set \( S \) is given by \( |S| \).
References
Capobianco, E.: High-dimensional role of AI and machine learning in cancer research. Br. J. Cancer 126, 523–532 (2022)
Bellman, R.E.: Adaptive Control Processes: A Guided Tour. Princeton University Press, Princeton (1961)
Bishop, C.M.: Pattern Recognition and Machine Learning (Information Science and Statistics). Springer-Verlag, Berlin, Heidelberg (2006)
Farmanbar, M., Toygar, Ö.: Feature selection for the fusion of face and palmprint biometrics. Signal Image Video Process. 10, 951–958 (2016)
Guyon, I., Elisseeff, A.: An introduction to variable and feature selection 1157–1182 (2003)
Lajevardi, S.M., Hussain, Z.M.: Automatic facial expression recognition: feature extraction and selection. Signal Image Video Process. 6, 159–169 (2012)
Peng, H., Long, F., Ding, C.: Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy. IEEE Trans. Pattern Anal. Mach. Intell. 27, 1226–1238 (2005)
Kira, K., Rendell, L.A.: In: Sleeman, D., Edwards, P. (eds.) Machine Learning Proceedings 1992, pp. 249–256. Morgan Kaufmann, San Francisco (CA) (1992)
Rida, I., Almaadeed, S., Bouridane, A.: Gait recognition based on modified phase-only correlation. Signal Image Video Process. 10, 463–470 (2016)
Tibshirani, R.: Regression shrinkage and selection via the Lasso. J. Royal Stat. Soc. Ser. B (Methodological) 58, 267–288 (1996). (Full publication date: 1996)
Breiman, L.: Random forests. Mach. Learn. 45, 5–32 (2001)
Saeys, Y., Inza, I., Larrañaga, P.: A review of feature selection techniques in bioinformatics. Bioinformatics 23, 2507–2517 (2007)
Atan, O., Zame, W.R., Feng, Q., van der Schaar, M.: Constructing effective personalized policies using counterfactual inference from biased data sets with many features. Mach. Learn. 108, 945–970 (2019)
Subasi, A.: A decision support system for diagnosis of neuromuscular disorders using DWT and evolutionary support vector machines. Signal Image Video Process. 9, 399–408 (2015)
Ruszczak, B., Smykała, K., Tomaszewski, M., Navarro Lorente, P.J.: Various tomato infection discrimination using spectroscopy. Signal Image Video Process. 2024
Chen, T., Guestrin, C.: Xgboost: A scalable tree boosting system. In: Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pp. 785–794 (2016)
Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press (2016)
Junejo, I.N., Bhutta, A.A., Foroosh, H.: Single-class SVM for dynamic scene modeling. Signal Image Video Process. 7, 45–52 (2013)
Aguiar, G., Krawczyk, B., Cano, A.: A survey on learning from imbalanced data streams: taxonomy, challenges, empirical study, and reproducible experimental framework. Mach. Learn. (2023)
Zhang, N., Chen, M., Yang, F., Yang, C., Yang, P., Gao, Y., Shang, Y., Peng, D.: Forest height map** using feature selection and machine learning by integrating multi-source satellite data in Baoding City. North China. Remote Sens. 14 (2022)
Sun, C., Dai, R.: Distributed optimization for convex mixed-integer programs based on projected subgradient algorithm. IEEE Conference on Decision and Control (CDC) 2018, 2581–2586 (2018)
Makridakis, S., Spiliotis, E., Assimakopoulos, V.: The M4 competition: 100,000 time series and 61 forecasting methods. Int. J. Forecast. 36, 54–74 (2020)
Akbilgic, O.: ISTANBUL STOCK EXCHANGE. UCI Machine Learning Repository (2013). https://doi.org/10.24432/C54P4J
Fontanella, F.: DARWIN. UCI Machine Learning Repository (2022). https://doi.org/10.24432/C55D0K
Author information
Authors and Affiliations
Contributions
Mehmet Y. Turali: Conceptualization, Methodology, Software, Writing- original draft and revised paper. Mehmet E. Lorasdagi: Conceptualization, Software, Writing- original draft. Suleyman S. Kozat: Conceptualization, Writing- review & editing.
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no known competing financial interests or personal relationships that could have influenced influence the work reported in this paper.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Turali, M.Y., Lorasdagi, M.E. & Kozat, S.S. AFS-BM: enhancing model performance through adaptive feature selection with binary masking. SIViP (2024). https://doi.org/10.1007/s11760-024-03411-x
Received:
Revised:
Accepted:
Published:
DOI: https://doi.org/10.1007/s11760-024-03411-x