SpringerLink
Log in

Machine learning with a reject option: a survey

  • Published:
Machine Learning Aims and scope Submit manuscript

Abstract

Machine learning models always make a prediction, even when it is likely to be inaccurate. This behavior should be avoided in many decision support applications, where mistakes can have severe consequences. Albeit already studied in 1970, machine learning with rejection recently gained interest. This machine learning subfield enables machine learning models to abstain from making a prediction when likely to make a mistake. This survey aims to provide an overview on machine learning with rejection. We introduce the conditions leading to two types of rejection, ambiguity and novelty rejection, which we carefully formalize. Moreover, we review and categorize strategies to evaluate a model’s predictive and rejective quality. Additionally, we define the existing architectures for models with rejection and describe the standard techniques for learning such models. Finally, we provide examples of relevant application domains and show how machine learning with rejection relates to other machine learning research areas.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Availability of data and material

Not applicable.

Code availability

Not applicable.

References

  • Abbas, M. R., Nadeem, M. S. A., Shaheen, A., Alshdadi, A. A., Alharbey, R., Shim, S.-O., & Aziz, W. (2019). Accuracy rejection normalized-cost curves (ARNCCs): A novel 3-dimensional framework for robust classification. IEEE Access, 7, 160125–160143.

    Google Scholar 

  • Abbasi, A., Albrecht, C., Vance, A., & Hansen, J. (2012). Metafraud: A meta-learning framework for detecting financial fraud. MIS Quarterly: Management Information Systems, 36(4), 1293–1327.

    Google Scholar 

  • Ahmadi, A., Omatu, S., Fu**aka, T., & Kosaka, T. (2004). Improvement of reliability in banknote classification using reject option and local PCA. Information Sciences, 168(1–4), 277–293.

    Google Scholar 

  • Alhoniemi, E., Himberg, J., & Vesanto, J. (1999). Probabilistic measures for responses of self-organizing map units. In H. Bothe, E. Oja, E. Massad and C. Haefke, (eds.), Proceedings of the international ICSC congress on computational intelligence methods and applications. ICSC Academic Press, Zurich, Switzerland, (pp. 286–290). ICSC Academic Press.

  • Amin, K., DeSalvo, G., & Rostamizadeh, A. (2021). Learning with labeling induced abstentions. Advances in Neural Information Processing Systems, 15(NeurIPS), 12576–12586.

    Google Scholar 

  • Arlandis, J., Perez-Cortes, J. C., & Cano, J. (2002). Rejection strategies and confidence measures for a k-NN classifier in an OCR task. In International Conference on Pattern Recognition, (vol. 16, pp. 576–579).

  • Asif, A., & Minhas, F. u. A. A. (2020). Generalized Neural Framework for Learning with Rejection. In 2020 International Joint Conference on Neural Networks (IJCNN), (pp. 1–8). IEEE.

  • Balsubramani, A. (2016). Learning to abstain from binary prediction, (pp. 1–23). ar**v preprintar**v:1602.08151

  • Barandas, M., Folgado, D., Santos, R., Simão, R., & Gamboa, H. (2022). Uncertainty-based rejection in machine learning: Implications for model development and interpretability. Electronics (Switzerland), 11(3), 1–20.

    Google Scholar 

  • Bartlett, P. L., & Wegkamp, M. H. (2008). Classification with a reject option using a hinge loss. Journal of Machine Learning Research, 9, 1823–1840.

    MathSciNet  Google Scholar 

  • Berlemont, S., Lefebvre, G., Duffner, S., & Garcia, C. (2015). Siamese neural network based similarity metric for inertial gesture classification and rejection. In 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), (pp. 1–6). IEEE.

  • Bock, P. (1988). A perspective on artificial intelligence: Learning to learn. Annals of Operations Research, 16(1), 33–52.

    Google Scholar 

  • Boulegane, D., Bifet, A., & Madhusudan, G. (2019). Arbitrated dynamic ensemble with abstaining for time-series forecasting on data streams. In 2019 IEEE international conference on big data (big data), (pp. 1040–1045). IEEE.

  • Bounsiar, A., Beauseroy, P., & Grall-Maës, E. (2008). General solution and learning method for binary classification with performance constraints. Pattern Recognition Letters, 29(10), 1455–1465.

    Google Scholar 

  • Brazdil, P., Carrier, C. G., Soares, C., & Vilalta, R. (2009). Metalearning—Applications to Data Mining. Springer Science & Business Media.

    Google Scholar 

  • Brinkrolf, J., & Hammer, B. (2017). Probabilistic extension and reject options for pairwise LVQ. In 2017 12th international workshop on self-organizing maps and learning vector quantization, clustering and data visualization (WSOM), (pp. 1–8). IEEE.

  • Brinkrolf, J., & Hammer, B. (2018). Interpretable machine learning with reject option. Automatisierungstechnik, 66(4), 283–290.

    Google Scholar 

  • Cao, Y., Cai, T., & An, B. (2022). Generalizing consistent multi-class classification with rejection to be compatible with arbitrary losses. (NeurIPS), (pp. 1–14).

  • Cardoso, J. S., & Pinto Da Costa, J. F. (2007). Learning to classify ordinal data: The data replication method. Journal of Machine Learning Research, 8, 1393–1429.

    MathSciNet  Google Scholar 

  • Cecotti, H., & Vajda, S. (2013). Rejection schemes in multi-class classification—Application to handwritten character recognition. In 2013 12th international conference on document analysis and recognition, (pp. 445–449). IEEE.

  • Charoenphakdee, N., Cui, Z., Zhang, Y., & Sugiyama, M. (2021). Classification with rejection based on cost-sensitive classification. In Proceedings of the 38th international conference on machine learning, (pp. 1507–1517).

  • Chen, B., Chen, B. F., & Lin, H. T. (2018). Rotation-blended CNNs on a new open dataset for tropical cyclone image-to-intensity regression. Proceedings of the ACM SIGKDD international conference on knowledge discovery and data mining, (pp. 90–99).

  • Chow, C. (1970). On optimum recognition error and reject tradeoff. IEEE Transactions on Information Theory, 16(1), 41–46.

    Google Scholar 

  • Clertant, M., Sokolovska, N., Chevaleyre, Y., & Hanczar, B. (2020). Interpretable cascade classifiers with abstention. In AISTATS 2019 - 22nd international conference on artificial intelligence and statistics.

  • Coenen, L., Abdullah, A. K., & Guns, T. (2020). Probability of default estimation, with a reject option. In 2020 IEEE 7th international conference on data science and advanced analytics (DSAA), (pp. 439–448). IEEE.

  • Cohen, K. M., Park, S., Simeone, O., & Shamai, S. (2022). Bayesian active meta-learning for reliable and efficient AI-based demodulation. IEEE Transactions on Signal Processing, 70, 5366–5380.

    MathSciNet  Google Scholar 

  • Condessa, F., Bioucas-Dias, J., Castro, C. A., Ozolek, J., & Kovacevic, J. (2013). Classification with reject option using contextual information. In 2013 IEEE 10th international symposium on biomedical imaging, (pp. 1340–1343), San Francisco, California. IEEE.

  • Condessa, F., Bioucas-Dias, J., Castro, C. A., Ozolek, J., & Kovačević, J. (2015a). Image classification with rejection using contextual information, (pp. 1–21). ar**v preprintar**v:1509.01287

  • Condessa, F., Bioucas-Dias, J., & Kovacevic, J. (2015b). Robust hyperspectral image classification with rejection fields. In Workshop on hyperspectral image and signal processing, evolution in remote sensing, (vol. 2015-June, pp. 1–4), Tokyo, Japan. IEEE.

  • Condessa, F., Bioucas-Dias, J., & Kovacevic, J. (2015). Supervised hyperspectral image classification with rejection. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 9(6), 2321–2332.

    Google Scholar 

  • Condessa, F., Bioucas-Dias, J., & Kovačević, J. (2017). Performance measures for classification systems with rejection. Pattern Recognition, 63, 437–450.

    Google Scholar 

  • Conte, D., Foggia, P., Percannella, G., Saggese, A., & Vento, M. (2012). An ensemble of rejecting classifiers for anomaly detection of audio events. In Proceedings - 2012 IEEE 9th international conference on advanced video and signal-based surveillance, AVSS 2012, (pp. 76–81). IEEE.

  • Corbière, C., Thome, N., Saporta, A., Vu, T. H., Cord, M., & Pérez, P. (2022). Confidence estimation via auxiliary models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10), 6043–6055.

    Google Scholar 

  • Cordelia, L. P., Foggia, P., Sansone, C., Tortorella, F., & Vento, M. (1998). Optimizing the error/reject trade-off for a multi-expert system using the Bayesian combining rule. In lecture notes in computer science (including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics), (vol. 1451, pp. 717–725).

  • Cordella, L., De Stefano, C., Fontanella, F., & Scotto Di Freca, A. S. (2014). Random forest for reliable pre-classification of handwritten characters. In Proceedings—international conference on pattern recognition, (pp. 1319–1324). IEEE.

  • Cordella, L., De Stefano, C., Sansone, C., & Vento, M. (1995). An adaptive reject option for LVQ classifiers. Lecture notes in computer science (vol. 974, pp. 68–73). Springer.

  • Cordella, L., De Stefano, C., Tortorella, F., & Vento, M. (1995). A method for improving classification reliability of multilayer perceptrons. IEEE Transactions on Neural Networks, 6(5), 1140–1147.

    Google Scholar 

  • Cortes, C., DeSalvo, G., Gentile, C., Mohri, M., & Yang, S. (2018). Online Learning with Abstention. In Proceedings of the 35th international conference on machine learning, volume 80 of Proceedings of machine learning research, (pp. 1059–1067), Stockholmsmässan, Stockholm Sweden. PMLR.

  • Cortes, C., DeSalvo, G., & Mohri, M. (2016a). Boosting with abstention. In Advances in neural information processing systems, (pp. 1668–1676).

  • Cortes, C., DeSalvo, G., & Mohri, M. (2016b). Learning with Rejection. In Proceedings of The 27th international conference on algorithmic learning theory (ALT 2016), volume 9925 LNAI, (pp. 67–82), Bari, Italy. Springer.

  • da Rocha Neto, A. R., Sousa, R., Barreto, G. A., & Cardoso, J. S. (2011). Diagnostic of pathology on the vertebral column with embedded reject option. In IbPRIA 2011: pattern recognition and image analysis, (vol. 6669 LNCS, pp. 588–595).

  • Dalitz, C. (2009). Reject options and confidence measures for kNN classifiers. Schriftenreihe des Fachbereichs Elektrotechnik und Informatik Hochschule Niederrhein, 8(2009), 16–38.

    Google Scholar 

  • De, A., Okati, N., Zarezade, A., & Rodriguez, M. G. (2021). Classification Under Human Assistance. In 35th AAAI conference on artificial intelligence, AAAI 2021, (vol. 7, pp. 5905–5913).

  • De Stefano, C., Sansone, C., & Vento, M. (2000). To reject or not to reject: That is the question-an answer in case of neural classifiers. IEEE Transactions on Systems, Man and Cybernetics, Part C (Applications and Reviews), 30(1), 84–94.

    Google Scholar 

  • Denis, C., & Hebiri, M. (2020). Consistency of plug-in confidence sets for classification in semi-supervised learning. Journal of Nonparametric Statistics, 32(1), 42–72.

    MathSciNet  Google Scholar 

  • Devos, L., Meert, W., & Davis, J. (2021). Versatile verification of tree ensembles. Proceedings of Machine Learning Research, 139, 2654–2664.

    Google Scholar 

  • Devos, L., Perini, L., Meert, W., & Davis, J. (2023). Detecting evasion attacks in deployed tree ensembles. In Joint European conference on machine learning and knowledge discovery in databases, (pp. 227–243).

  • Du, J., Ni, E. A., & Ling, C. X. (2010). Adapting cost-sensitive learning for reject option. In International conference on information and knowledge management, proceedings, (pp. 1865–1868).

  • Dubos, C., Bernard, S., Adam, S., & Sabourin, R. (2016). ROC-based cost-sensitive classification with a reject option. In 2016 23rd international conference on pattern recognition (ICPR), (pp. 3320–3325). IEEE.

  • Dubuisson, B., & Masson, M. (1993). A statistical decision rule with incomplete knowledge about classes. Pattern Recognition, 26(1), 155–165.

    Google Scholar 

  • Dübuisson, B., Usai, M., & Malvache, P. (1985). Computer aided system diagnostic with an incomplete learning set. Progress in Nuclear Energy, 15(C), 875–880.

    Google Scholar 

  • El-Yaniv, R., & Wiener, Y. (2010). On the foundations of noise-free selective classification. Journal of Machine Learning Research, 11, 1605–1641.

    MathSciNet  Google Scholar 

  • El-Yaniv, R., & Wiener, Y. (2011). Agnostic selective classification. Advances in Neural Information Processing Systems, 24, 1–9.

    Google Scholar 

  • Feng, L., Ahmed, M. O., Hajimirsadeghi, H., & Abdi, A. (2022). Towards Better Selective Classification. In International conference on learning and representation.

  • Ferri, C., Flach, P., & Hernández-Orallo, J. (2004). Delegating classifiers. Proceedings, twenty-first international conference on machine learning, ICML, 2004, (pp. 289–296).

  • Ferri, C., & Hernández-Orallo, J. (2004). Cautious lassifiers. Proceedings of ROC analysis in artificial intelligence, 1st international workshop (ROCAI-2004), (pp. 27–36).

  • Filchenkov, A., & Pendryak, A. (2016). Datasets meta-feature description for recommending feature selection algorithm. In Proceedings of artificial intelligence and natural language and information extraction, social media and web search FRUCT conference, AINL-ISMW FRUCT, 2015, (vol. 7, pp. 11–18).

  • Fischer, L., Hammer, B., & Wersing, H. (2014a). Local rejection strategies for learning vector quantization. In Lecture notes in computer science, volume 8681 LNCS, (pp. 563–570). Springer.

  • Fischer, L., Hammer, B., & Wersing, H. (2015). Efficient rejection strategies for prototype-based classification. Neurocomputing, 169, 334–342.

    Google Scholar 

  • Fischer, L., Hammer, B., & Wersing, H. (2015b). Optimum reject options for prototype-based classification. ar**v preprintar**v:1503.06549.

  • Fischer, L., Hammer, B., & Wersing, H. (2016). Optimal local rejection for classifiers. Neurocomputing, 214, 445–457.

    Google Scholar 

  • Fischer, L., Nebel, D., Villmann, T., Hammer, B., & Wersing, H. (2014b). Rejection strategies for learning vector quantization - a comparison of probabilistic and deterministic approaches. In 22nd European symposium on artificial neural networks, computational intelligence and machine learning, ESANN 2014 - roceedings, (pp. 109–118). Springer.

  • Fischer, L. & Villmann, T. (2016). A probabilistic classifier model with adaptive rejection option. http://www.techfak.uni-bielefeld.de/~fschleif/mlr/mlr.html.

  • Flores, I. (1958). An optimum character recognition system using decision functions. IRE Transactions on Electronic Computers, EC–7(2), 180.

    Google Scholar 

  • Franc, V., & Prusa, D. (2019). On discriminative learning of prediction uncertainty. In 36th international conference on machine learning, ICML 2019, 2019-June, (pp. 3465–3480).

  • Franc, V., Prusa, D., & Voracek, V. (2021). Optimal strategies for reject option classifiers. Journal of Machine Learning Research, 24, 1–49.

    Google Scholar 

  • Frélicot, C. (1997). Learning rejection thresholds for a class of fuzzy classifiers from possibilistic clustered noisy data. In 7th International fuzzy systems association world congress, IFSA, (vol. 3, pp. 111–116).

  • Frélicot, C. (1998). On unifying probabilistic/fuzzy and possibilistic rejection-based classifiers. SSPR /SPR 1998: Advances in Pattern Recognition (pp. 736–745). Springer.

  • Frélicot, C., & Mascarilla, L. (2002). Reject strategies driven combination of pattern classifiers. Pattern Analysis and Applications, 5(2), 234–243.

    MathSciNet  Google Scholar 

  • Friedel, C. C., Rückert, U., & Kramer, S. (2006). Cost Curves for Abstaining Classifiers. In ICML 2006 - 3th workshop ROC analysis in ML, Pittsburgh, Pennsylvania.

  • Fu, Y., Zhu, X., & Li, B. (2013). A survey on instance selection for active learning. Knowledge and Information Systems, 35(2), 249–283.

    Google Scholar 

  • Fukunaga, K., & Kessell, D. (1972). Application of optimum error-reject functions. IEEE Transactions on Information Theory, 18(6), 814–817.

    Google Scholar 

  • Fumera, G., Pillai, I., & Roli, F. (2003). Classification with reject option in text categorisation systems. In 12th international conference on image analysis and processing, 2003. Proceedings, (pp. 582–587). IEEE Comput. Soc.

  • Fumera, G., Pillai, I., & Roli, F. (2004). A two-stage classifier with reject option for text categorisation. Lecture Notes in Computer Science (Vol. 3138, pp. 771–779). Springer.

  • Fumera, G., & Roli, F. (2002). Support vector machines with embedded reject option. In Pattern recognition with support vector machines (pp. 68–82). Springer.

  • Fumera, G., & Roli, F. (2004). Analysis of error-reject trade-off in linearly combined multiple classifiers. Pattern Recognition, 37(6), 1245–1265.

    Google Scholar 

  • Fumera, G., Roli, F., & Giacinto, G. (2000). Multiple reject thresholds for improving classification reliability. In Lecture notes in computer science, volume 1876 LNCS (pp. 863–871). Springer.

  • Gal, Y., Hron, J., & Kendall, A. (2017). Concrete Dropout. In Advances in neural information processing systems, (vol. 30, pp. 3581–3590). Curran Associates, Inc.

  • Gamelas Sousa, R., da Rocha-Neto, A. R., Cardoso, J. S., & Barreto, G. A. (2015). Robust classification with reject option using the self-organizing map. Neural Computing and Applications, 26(7), 1603–1619.

    Google Scholar 

  • Gamelas Sousa, R., da Rocha Neto, A. R., Barreto, G. A., Cardoso, J. S., & Coimbra, M. T. (2014a). Reject option paradigm for the reduction of support vectors. 22nd European symposium on artificial neural networks, computational intelligence and machine learning, ESANN 2014 - Proceedings, (pp. 331–336).

  • Gamelas Sousa, R., da Rocha Neto, A. R., Cardoso, J. S., & Barreto, G. A. (2014). Classification with reject option using the self-organizing map. In Lecture notes in computer science, volume 8681 LNCS, chapter Artificial, (pp. 105–112). Springer.

  • Gamelas Sousa, R., Mora, B., & Cardoso, J. S. (2009). An ordinal data method for the classification with reject option. In 8th international conference on machine learning and applications, ICMLA 2009, (vol. 2, pp. 746–750).

  • Gangrade, A., Kag, A., Cutkosky, A., & Saligrama, V. (2021). Online selective classification with limited feedback. Advances in Neural Information Processing Systems, 18(NeurIPS), 14529–14541.

    Google Scholar 

  • Gangrade, A., Kag, A., & Saligrama, V. (2021b). Selective classification via one-sided prediction. In Proceedings of the 24th international conference on artificial intelligence and statistics (AISTATS), (vol. 130, p. 22), San Diego, California, USA. Proceedings of Machine Learning Research.

  • Gasca, A., & E., Saldaña T., S., Sánchez G., J. S., Velásquez G., V., Rendón L., E., Abundez B., I. M., Valdovinos R., R. M., & Cruz R., R. (2011). A rejection option for the multilayer perceptron using hyperplanes. In Lecture notes in computer science, (vol. 6593 LNCS, pp. 51–60). Springer.

  • Geifman, Y., & El-Yaniv, R. (2017). Selective classification for deep neural networks. In Advances in neural information processing systems, (pp. 4879–4888). Curran Associates, Inc.

  • Geifman, Y., & El-Yaniv, R. (2019). SelectiveNet: A deep neural network with an integrated reject option. In International conference on machine learning (ICML 2019), (vol. 2019-June, pp. 3768–3776).

  • Giraud-Carrier, C. (2022). Combining base-learners into ensembles. Springer International Publishing.

  • Glodek, M., Schels, M., Palm, G., & Schwenker, F. (2012). Multiple classifier combination using reject options and markov fusion networks. In ICMI’12 - Proceedings of the ACM international conference on multimodal interaction, (pp. 465–472).

  • Golfarelli, M., Maio, D., & Malton, D. (1997). On the error-reject trade-off in biometric verification systems. IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(7), 786–796.

    Google Scholar 

  • Göpfert, J. P., Hammer, B., & Wersing, H. (2018). Mitigating concept drift via rejection. In Lecture notes in computer science, volume 11139 LNCS, (pp. 456–467). Springer.

  • Grandvalet, Y., Rakotomamonjy, A., Keshet, J., & Canu, S. (2009). Support vector machines with a reject option. In Advances in neural information processing systems, (vol. 21, pp. 537–544). Curran Associates, Inc.

  • Gridin, I. (2022). Hyperparameter Optimization.

  • Guan, H., Zhang, Y., Cheng, H. D., & Tang, X. (2020). Bounded-abstaining classification for breast tumors in imbalanced ultrasound images. International Journal of Applied Mathematics and Computer Science, 30(2), 325–336.

    MathSciNet  Google Scholar 

  • Hamid, K., Asif, A., Abbasi, W., Sabih, D., & Afsar, F. A. (2017). Machine learning with abstention for automated liver disease diagnosis. In 2017 international conference on frontiers of information technology (FIT), (vol. 2017-Jan, pp. 356–361). IEEE.

  • Hanczar, B. (2019). Performance visualization spaces for classification with rejection option. Pattern Recognition, 96, 106984.

    Google Scholar 

  • Hanczar, B., & Dougherty, E. R. (2008). Classification with reject option in gene expression data. Bioinformatics, 24(17), 1889–1895.

    Google Scholar 

  • Hanczar, B., & Sebag, M. (2014). Combination of one-class support vector machines for classification with reject option. Machine Learning and Knowledge Discovery in Databases - Part I (pp. 547–562). Springer.

  • Hansen, L. K., Liisberg, C., & Salamon, P. (1997). The error-reject tradeoff. Open Systems & Information Dynamics, 4(2), 159–184.

    Google Scholar 

  • Hatami, N., & Chira, C. (2013). Classifiers with a reject option for early time-series classification. Proceedings of the 2013 IEEE symposium on computational intelligence and ensemble learning, CIEL 2013 - 2013 IEEE symposium series on computational intelligence, SSCI 2013, (pp. 9–16).

  • Hellman, M. E. (1970). The nearest neighbor classification rule with a reject option. IEEE Transactions on Systems Science and Cybernetics, 6(3), 179–185.

    Google Scholar 

  • Hendrickx, K., Meert, W., Cornelis, B., & Davis, J. (2022). Know your limits: Machine learning with rejection for vehicle engineering. In Lecture notes in computer science (including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics), (vol. 13087 LNAI, pp. 273–288).

  • Heo, J., Lee, H. B., Kim, S., Lee, J., Kim, K. J., Yang, E., & Hwang, S. J. (2018). Uncertainty-aware attention for reliable interpretation and prediction. In Advances in neural information processing systems, (vol. 31, pp. 909–918), Montreal, Canada. Curran Associates, Inc.

  • Herbei, R., & Wegkamp, M. H. (2006). Classification with reject option. Canadian Journal of Statistics, 34(4), 709–721.

    MathSciNet  Google Scholar 

  • Homenda, W., Luckner, M., & Pedrycz, W. (2014). Classification with rejection based on various SVM techniques. In 2014 International joint conference on neural networks (IJCNN), (pp. 3480–3487). IEEE.

  • Homenda, W., Luckner, M., & Pedrycz, W. (2016). Classification with rejection: Concepts and evaluations. Advances in Intelligent Systems and Computing, 364(December), 413–425.

    Google Scholar 

  • Hosseini, H., Chen, Y., Kannan, S., Zhang, B., & Poovendran, R. (2017). Blocking transferability of adversarial examples in black-box learning systems. ar**v preprintar**v:1703.04318.

  • Hsu, Y. C., Shen, Y., **, H., & Kira, Z. (2020). Generalized ODIN: Detecting out-of-distribution image without learning from out-of-distribution data. In Proceedings of the IEEE computer society conference on computer vision and pattern recognition, (pp. 10948–10957).

  • Huang, L., Zhang, C., & Zhang, H. (2020). Self-adaptive training: Beyond empirical risk minimization. Advances in neural information processing systems, 2020-Decem(NeurIPS), (pp. 1–12).

  • Huang, Y., & Suen, C. (1995). A method of combining multiple experts for the recognition of unconstrained handwritten numerals. IEEE Transactions on Pattern Analysis and Machine Intelligence, 17(1), 90–94.

    Google Scholar 

  • Jiang, W., Zhao, Y., & Wang, Z. (2020). Risk-controlled selective prediction for regression deep neural network models. In 2020 international joint conference on neural networks (IJCNN), (pp. 1–8). IEEE.

  • Jigang, X., Zhengding, Q., & Jie, W. (2006). Bootstrap methods for reject rules of fisher LDA. Proceedings - International Conference on Pattern Recognition, 3, 425–428.

    Google Scholar 

  • Kalai, A. T., & Kanade, V. (2021). Towards optimally abstaining from prediction with OOD test examples. Advances in Neural Information Processing Systems, 16(NeurIPS), 12774–12785.

    Google Scholar 

  • Kang, S., Cho, S., Rhee, S.-J., & Yu, K.-S. (2017). Reliable prediction of anti-diabetic drug failure using a reject option. Pattern Analysis and Applications, 20(3), 883–891.

    MathSciNet  Google Scholar 

  • Kato, M., Cui, Z., & Fukuhara, Y. (2020). ATRO: Adversarial training with a rejection option. 2019, (pp. 1–18).

  • Khodra, M. L. (2016). Delegating classifiers for automatic text categorization delegating classifiers for automatic text categorization (June).

  • Kocak, M. A., Ramirez, D., Erkip, E., & Shasha, D. (2020). SafePredict: A meta-algorithm for machine learning that uses refusals to guarantee correctness. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(2), 1.

    Google Scholar 

  • Kompa, B., Snoek, J., & Beam, A. L. (2021). Second opinion needed: Communicating uncertainty in medical machine learning. NPJ Digital Medicine, 4(1), 4.

    Google Scholar 

  • Korycki, L., Cano, A., & Krawczyk, B. (2019). Active learning with abstaining classifiers for imbalanced drifting data streams. In 2019 IEEE international conference on big data (big data), (pp. 2334–2343). IEEE.

  • Kotelevskii, N., Artemenkov, A., Fedyanin, K., Noskov, F., Fishkov, A., Shelmanov, A., Vazhentsev, A., Petiushko, A., & Panov, M. (2022). Nonparametric uncertainty quantification for single deterministic neural network. (NeurIPS 2022), (pp. 1–16).

  • Kotropoulos, C., & Arce, G. R. (2009). Linear classifier with reject option for the detection of vocal fold paralysis and vocal fold edema. Eurasip Journal on Advances in Signal Processing, 2009.

  • Krawczyk, B., & Cano, A. (2018). Online ensemble learning with abstaining classifiers for drifting and noisy data streams. Applied Soft Computing, 68, 677–692.

    Google Scholar 

  • Kühne, J., März, C., & Gühmann, C. (2021). Securing deep learning models with autoencoder based anomaly detection. In Proceedings of the European conference of the PHM society 2021, (pp. 221–233), Virtual event. PHM Society.

  • Kummert, J., Paassen, B., Jensen, J., Gopfert, C., & Hammer, B. (2016). Local reject option for deterministic multi-class SVM. In Artificial neural networks and machine learning–ICANN 2016: 25th international conference on artificial neural networks, Barcelona, Spain, September 6-9, 2016, Proceedings, Part II 25, (pp. 251–258), Springer.

  • Kwok, J. T. (1999). Moderating the outputs of support vector machine classifiers. IEEE Transactions on Neural Networks, 10(5), 1018–1031.

    Google Scholar 

  • Lam, L., & Suen, C. Y. (1995). Optimal combinations of pattern classifiers. Pattern Recognition Letters, 16(9), 945–954.

    Google Scholar 

  • Landgrebe, T. C., Tax, D. M., Paclík, P., & Duin, R. P. (2006). The interaction between classification and reject performance for distance-based reject-option classifiers. Pattern Recognition Letters, 27(8), 908–917.

    Google Scholar 

  • Landgrebe, T. C., Tax, D. M., Paclík, P., Duin, R. P., & Andrew, C. (2004). A combining strategy for ill-defined problems. In Proceedings of the 15th annual symposium of the pattern recognition association of South Africa, (pp. 57–62).

  • Laroui, S., Descombes, X., Vernay, A., Villiers, F., Debreuve, E., Laroui, S., Descombes, X., Vernay, A., Villiers, F., Villalba, F., & Laroui, S. (2021). How to define a rejection class based on model learning? In 25th international conference on pattern recogniton, Milano.

  • Le Capitaine, H., & Frélicot, C. (2012). A family of measures for best top-n class-selective decision rules. Pattern Recognition, 45(1), 552–562.

    Google Scholar 

  • Lee, J. K., Bu, Y., Rajan, D., Sattigeri, P., Panda, R., Das, S., & Wornell, G. W. (2021). Fair selective classification via sufficiency. In Proceedings of the 38th international conference on machine learning, (vol. 139, pp. 6076–6086).

  • Lei, J. (2014). Classification with confidence. Biometrika, 101(4), 755–769.

    MathSciNet  Google Scholar 

  • Lewicke, A., Sazonov, E. S., Corwin, M., Neuman, M., & Schuckers, S. (2008). Sleep versus wake classification from heart rate variability using computational intelligence: Consideration of rejection in classification models. IEEE Transactions on Biomedical Engineering, 55(1), 108–118.

    Google Scholar 

  • Li, M., & Sethi, I. K. (2006). Confidence-based classifier design. Pattern Recognition, 39(7), 1230–1240.

    Google Scholar 

  • Lin, D., Sun, L., Toh, K. A., Zhang, J. B., & Lin, Z. (2018). Twin SVM with a reject option through ROC curve. Journal of the Franklin Institute, 355(4), 1710–1732.

    MathSciNet  Google Scholar 

  • Lin, Z., Glass, L., Westover, M. B., **ao, C., & Sun, J. (2022). SCRIB: Set-Classifier with class-specific risk bounds for blackbox models. Proceedings of the AAAI Conference on Artificial Intelligence, 36(7), 7497–7505.

    Google Scholar 

  • Liu, J., Gallego, B., & Barbieri, S. (2022). Incorporating uncertainty in learning to defer algorithms for safe computer-aided diagnosis. Scientific Reports, 12(1), 1–9.

    Google Scholar 

  • Loeffel, P. X., Marsala, C., & Detyniecki, M. (2015). Classification with a reject option under Concept Drift: The Droplets algorithm. In Proceedings of the 2015 IEEE international conference on data science and advanced analytics, DSAA 2015, (pp. 1–9).

  • Lotte, F., Mouchère, H., & Lécuyer, A. (2008). Pattern rejection strategies for the design of self-paced EEG-based brain-computer interfaces. Proceedings - international conference on pattern recognition, (pp. 6–10).

  • Lou, Z., Liu, K., Yang, J., & Suen, C. (1999). Rejection criteria and pairwise discrimination of handwritten numerals based on structural features. Pattern Analysis and Applications, 2(3), 228–238.

    Google Scholar 

  • Ma, C., Randolph, M. A., & Drish, J. (2001). A support vector machines-based rejection technique for speech recognition. ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings, 1, 381–384.

    Google Scholar 

  • Madras, D., Pitassi, T., & Zemel, R. (2018). Predict responsibly: Improving fairness and accuracy by learning to defer. Advances in Neural Information Processing Systems, 2018(NeurIPS), 6147–6157.

  • Markou, M., & Singh, S. (2003). Novelty detection: A review-part 1: Statistical approaches. Signal Processing, 83(12), 2481–2497.

    Google Scholar 

  • Marrocco, C., Molinara, M., & Tortorella, F. (2007). An empirical comparison of ideal and empirical ROC-based reject rules. Lecture notes in computer science, volume 4571 LNAI (pp. 47–60). Springer.

  • Martens, T., Perini, L., & Davis, J. (2023). Semi-supervised learning from active noisy soft labels for anomaly detection. In Joint European conference on machine learning and knowledge discovery in databases. Springer.

  • Mena, J., Pujol, O., & Vitria, J. (2020). Uncertainty-based rejection wrappers for black-box classifiers. IEEE Access, 8, 101721–101746.

    Google Scholar 

  • Mesquita, D. P., Rocha, L. S., Gomes, J. P. P., & da Rocha Neto, A. R. (2016). Classification with reject option for software defect prediction. Applied Soft Computing Journal, 49, 1085–1093.

    Google Scholar 

  • Mozannar, H., & Sontag, D. (2020). Consistent estimators for learning to defer to an expert. ar**v preprintar**v:2006.01862.

  • Muzzolini, R., Yang, Y.-H., & Pierson, R. (1998). Classifier design with incomplete knowledge. Pattern Recognition, 31(4), 345–369.

    Google Scholar 

  • Nadeem, M. S. A., Zucker, J., & Hanczar, B. (2010). Accuracy-rejection curves (ARCs) for comparing classification methods with a reject option. Machine Learning in Systems Biology, 8, 65–81.

    Google Scholar 

  • Nalisnick, E., Matsukawa, A., Whye, Y., Dilan, T., & Balaji, G. (2019). Hybrid models with deep and invertible features (ICML, 2019).

  • Nguyen, V.-L., Destercke, S., & Hüllermeier, E. (2019). Epistemic uncertainty sampling. In International conference on discovery science DS 2019, volume 11828 LNAI, (pp. 72–86). Springer.

  • Nguyen, V. L., & Hüllermeier, E. (2020). Reliable multilabel classification: Prediction with partial abstention. In AAAI 2020 - 34th AAAI conference on artificial intelligence, (vol. 34, pp. 5264–5271).

  • Nguyen, V. L., & Hüllermeier, E. (2021). Multilabel classification with partial Abstention: Bayes-optimal prediction under label independence. Journal of Artificial Intelligence Research, 72, 613–665.

    MathSciNet  Google Scholar 

  • Ni, C., Charoenphakdee, N., Honda, J., & Sugiyama, M. (2019). On the calibration of multiclass classification with rejection, (pp. 1–31). ar**v preprintar**v:1901.10655

  • Pang, T., Zhang, H., He, D., Dong, Y., Su, H., Chen, W., Zhu, J., & Liu, T.-Y. (2021). Adversarial training with rectified rejection, (pp. 1–23). ar**v preprintar**v:2105.14785

  • Pang, T., Zhang, H., He, D., Dong, Y., Su, H., Chen, W., Zhu, J., & Liu, T. Y. (2022). Two coupled rejection metrics can tell adversarial examples apart. In Proceedings of the IEEE computer society conference on computer vision and pattern recognition, volume 2022-June, (pp. 15202–15212).

  • Pazzani, M. J., Murphy, P., Ali, K., & Schulenburg, D. (1994). Tradingoff coverage for accuracy in forecasts: Applications to clinical data analysis. In Proceedings of the AAAI symposium on AI in medicine, (pp. 100–104).

  • Perello-Nieto, M., Filho, T. M., Kull, M., & Flach, P. (2017). Background check: A general technique to build more reliable and versatile classifiers. In Proceedings - IEEE international conference on data mining, ICDM, (pp. 1143–1148).

  • Perini, L., Buerkner, P., & Klami, A. (2022a). Estimating the contamination factor’s distribution in unsupervised anomaly detection. In International conference on machine learning (ICML 2023).

  • Perini, L., Galvin, C., & Vercruyssen, V. (2020). A Ranking stability measure for quantifying the robustness of anomaly detection methods. Communications in Computer and Information Science, 1323, 397–408.

    Google Scholar 

  • Perini, L., Giannuzzi, D., & Davis, J. (2023). How to allocate your label budget? Choosing between active learning and learning to reject in anomaly detection. (1).

  • Perini, L., Vercruyssen, V., & Davis, J. (2020b). Quantifying the confidence of anomaly detectors in their example-wise predictions. In Proceedings of European conference on machine learning and principles and practice of knowledge discovery in databases, Ghent, Belgium. ECML-PKDD.

  • Perini, L., Vercruyssen, V., & Davis, J. (2022b). Transferring the contamination factor between anomaly detection domains by shape similarity. In Proceedings of the 36th AAAI conference on artificial intelligence, AAAI 2022, (vol. 36, pp. 4128–4136).

  • Pietraszek, T. (2005). Optimizing abstaining classifiers using ROC analysis. In Proceedings of the 22nd international conference on Machine learning - ICML ’05, (pp. 665–672), New York, New York, USA. ACM Press.

  • Pietraszek, T. (2007). On the use of ROC analysis for the optimization of abstaining classifiers. Machine Learning, 68(2), 137–169.

    Google Scholar 

  • Pillai, I., Fumera, G., & Roli, F. (2011). A classification approach with a reject option for multi-label problems. In Lecture notes in computer science, volume 6978 LNCS, (pp. 98–107). Springer.

  • Pillai, I., Fumera, G., & Roli, F. (2013). Multi-label classification with a reject option. Pattern Recognition, 46(8), 2256–2266.

    Google Scholar 

  • Pimentel, M. A., Clifton, D. A., Clifton, L., & Tarassenko, L. (2014). A review of novelty detection. Signal Processing, 99, 215–249.

    Google Scholar 

  • Prasad, M., & Sowmya, A. (2008). Multi-level classification of emphysema in HRCT lung images using delegated classifiers. Lecture notes in computer science (including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics), 5241 LNCS(PART 1), 59–66.

  • Prasad, N. R., Almanza-Garcia, S., & Lu, T. T. (2009). Anomaly detection. Computers, Materials and Continua, 14(1), 1–22.

    Google Scholar 

  • Puchkin, N., & Zhivotovskiy, N. (2022). Exponential savings in agnostic active learning through abstention. IEEE Transactions on Information Theory, 68(7), 4651–4665.

    MathSciNet  Google Scholar 

  • Pudil, P., Novovičova, J., Bláha, S., & Kittler, J. (1992). Multistage pattern recognition with reject option. In Proceedings - international conference on pattern recognition, (Vol. 2, pp. 92–95).

  • Pugnana, A., & Ruggieri, S. (2022). AUC-based selective classification. In International conference on artificial intelligence and statistics, 206.

  • Pugnana, A., & Ruggieri, S. (2023). A model-agnostic heuristics for selective classification. Proceedings of the AAAI Conference on Artificial Intelligence, 37(8), 9461–9469.

    Google Scholar 

  • Rahman, A., & Fairhurst, M. (1998). An evaluation of multi-expert configurations for the recognition of handwritten numerals. Pattern Recognition, 31(9), 1255–1273.

    Google Scholar 

  • Ramaswamy, H. G., Tewari, A., & Agarwal, S. (2018). Consistent algorithms for multiclass classification with an abstain option. Electronic Journal of Statistics, 12(1), 530–554.

    MathSciNet  Google Scholar 

  • Ruggieri, S., Alvarez, J. M., Pugnana, A., State, L., & Turini, F. (2023). Can We Trust Fair-AI ? AAAI 2023.

  • Sambu Seo, Wallat, M., Graepel, T., & Obermayer, K. (2000). Gaussian process regression: Active data selection and test point rejection. In Proceedings of the IEEE-INNS-ENNS international joint conference on neural networks. IJCNN 2000. Neural computing: New challenges and perspectives for the New Millennium, (Vol. 3, pp. 241–246). IEEE.

  • Sansone, C., Tortorella, F., & Vento, M. (2001). A classification reliability driven reject rule for multi-expert systems. International Journal of Pattern Recognition and Artificial Intelligence, 15(06), 885–904.

    Google Scholar 

  • Santos-Pereira, C. M., & Pires, A. M. (2005). On optimal reject rules and ROC curves. Pattern Recognition Letters, 26(7), 943–952.

    Google Scholar 

  • Sayedi, A., Zadimoghaddam, M., & Blum, A. (2010). Trading off mistakes and don’t-know predictions. Advances in neural information processing systems 23: 24th annual conference on neural information processing systems 2010, NIPS 2010, (pp. 1–9).

  • Senge, R., Bösner, S., Dembczyński, K., Haasenritter, J., Hirsch, O., Donner-Banzhoff, N., & Hüllermeier, E. (2014). Reliable classification: Learning classifiers that distinguish aleatoric and epistemic uncertainty. Information Sciences, 255, 16–29.

    MathSciNet  Google Scholar 

  • Settles, B. (2009). Active learning literature survey.

  • Sharma, M., & Bilgic, M. (2017). Evidence-based uncertainty sampling for active learning. Data Mining and Knowledge Discovery, 31(1), 164–202.

    MathSciNet  Google Scholar 

  • Shekhar, S., Ghavamzadeh, M., & Javidi, T. (2019). Binary classification with bounded abstention rate, (pp. 1–35). ar**v preprintar**v:1905.09561

  • Shekhar, S., Ghavamzadeh, M., & Javidi, T. (2020). Active learning for classification with abstention. In IEEE International Symposium on Information Theory - Proceedings, 2020-June(2), 2801–2806.

  • Shen, S. Q., Yang, B. B., & Gao, W. (2020a). AUC optimization with a reject option. In AAAI 2020 - 34th AAAI conference on artificial intelligence, (pp. 5686–5691).

  • Shen, Z., Chen, X., & Garibaldi, J. M. (2020b). A novel meta learning framework for feature selection using data synthesis and fuzzy similarity. In IEEE international conference on fuzzy systems, 2020-July.

  • Shpakova, T., & Sokolovska, N. (2021). Probabilistic personalised cascade with abstention. Pattern Recognition Letters, 147, 8–15.

    Google Scholar 

  • Singh, S., & Markou, M. (2004). An approach to novelty detection applied to the classification of image regions. IEEE Transactions on Knowledge and Data Engineering, 16(4), 396–407.

    Google Scholar 

  • Sotgiu, A., Demontis, A., Melis, M., Biggio, B., Fumera, G., Feng, X., & Roli, F. (2020). Deep neural rejection against adversarial examples. EURASIP Journal on Information Security, 2020(1), 5.

    Google Scholar 

  • Suutala, J., Pirttikangas, S., Riekki, J., & Röning, J. (2004). Reject-optional LVQ-based two-level classifier to improve reliability in footstep identification. In Lecture notes in computer science, (vol. 3001, pp. 182–187). Springer.

  • Tang, W., & Sazonov, E. S. (2014). Highly accurate recognition of human postures and activities through classification with rejection. IEEE Journal of Biomedical and Health Informatics, 18(1), 309–315.

    Google Scholar 

  • Tax, D. M., & Duin, R. P. (2008). Growing a multi-class classifier with a reject option. Pattern Recognition Letters, 29(10), 1565–1570.

    Google Scholar 

  • Temanni, M.-r., & Nadeem, S. A. (2007). Aggregating Abstaining and Delegating Classifiers For Improving Classification performance: An application to lung cancer survival prediction. Camda 07, (January 2017).

  • Thulasidasan, S., Bhattacharya, T., Bilmes, J., Chennupati, G., & Mohd-Yusof, J. (2019). Combating label noise in deep learning using abstention. In Proceedings of the 36th international conference on machine learning.

  • Tortorella, F. (2000). An optimal reject rule for binary classifiers. In Lecture notes in computer science, vol. 1876 LNCS, (pp. 611–620). Springer.

  • Tremmel, C., Fernandez-Vargas, J., Stamos, D., Cinel, C., Pontil, M., Citi, L., & Poli, R. (2022). A meta-learning BCI for estimating decision confidence. Journal of Neural Engineering, 19(4).

  • Ulmer, D., & Cinà, G. (2020). Know your limits: Uncertainty estimation with ReLU classifiers fails at reliable OOD detection. ar**v preprintar**v:2012.05329

  • Urahama, K., & Furukawa, Y. (1995). Gradient descent learning of nearest neighbor classifiers with outlier rejection. Pattern Recognition, 28(5), 761–768.

    Google Scholar 

  • Vailaya, A., & Jain, A. (2000). Reject option for VQ-based Bayesian classification. In Proceedings 15th international conference on pattern recognition. ICPR-2000, (vol. 2, pp. 48–51). IEEE Comput. Soc.

  • Van Craenendonck, T., Meert, W., Dumančić, S., & Blockeel, H. (2018). COBRASTS: A new approach to semi-supervised clustering of time series. Lecture notes in computer science (including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics), 11198 LNAI, 179–193.

  • Van der Plas, D., Meert, W., Verbraecken, J., & Davis, J. (2021). A reject option for automated sleep stage scoring. In ICML workshop on interpretable machine learning in healthcare, Virtual event.

  • Van der Plas, D., Meert, W., Verbraecken, J., & Davis, J. (2023). A novel reject option applied to sleep stage scoring. In Proceedings of the 2023 SIAM international conference on data mining (SDM), pp. 820–828.

  • Vanderlooy, S., Sprinkhuizen-Kuyper, I., & Smirnov, E. (2006a). Reliable classifiers in ROC space. Proceedings of the 15th BENELEARN machine learning conference, p. 27.

  • Vanderlooy, S., Sprinkhuizen-Kuyper, I. G., & Smirnov, E. N. (2006b). An analysis of reliable classifiers through ROC isometrics. ROC analysis in machine learning, (Citeseer).

  • Vanschoren, J. (2018). Meta-learning: A survey. pp. 1–29.

  • Varshney, K. R. (2006). A kernel based rejection method for supervised classification. International Journal of Computational Intelligence, 3(4), 312–321.

    Google Scholar 

  • Varshney, K. R. (2011). A risk bound for ensemble classification with a reject option. In 2011 IEEE statistical signal processing workshop (SSP), (pp. 769–772). IEEE.

  • Vasconcelos, G., Fairhurst, M., & Bisset, D. (1993). Enhanced reliability of multilayer perceptron networks through controlled pattern rejection. Electronics Letters, 29(3), 261.

    Google Scholar 

  • Vasconcelos, G., Fairhurst, M., & Bisset, D. (1995). Investigating feedforward neural networks with respect to the rejection of spurious patterns. Pattern Recognition Letters, 16(2), 207–212.

    Google Scholar 

  • Vercruyssen, V., Meert, W., Verbruggen, G., Maes, K., Baumer, R., & Davis, J. (2018). Semi-supervised anomaly detection with an application to water analytics. Proceedings - IEEE international conference on data mining, ICDM, 2018-Nov, (pp. 527–536).

  • Villmann, T., Kaden, M., Nebel, D., & Biehl, M. (2015). Learning vector quantization with adaptive cost-based outlier-rejection. In Lecture notes in computer science, (vol. 9257, pp. 772–782). Springer.

  • Wang, X., & Yiu, S. M. (2020). Classification with rejection: Scaling generative classifiers with supervised deep infomax. In Proceedings of the twenty-ninth international joint conference on artificial intelligence, (pp. 2980–2986), California. International joint conferences on artificial intelligence organization.

  • Wegkamp, M. H. (2007). Lasso type classifiers with a reject option. Electronic Journal of Statistics, 1, 155–168.

    MathSciNet  Google Scholar 

  • Wegkamp, M. H., & Yuan, M. (2012). Support vector machines with a reject option. Bernoulli, 17(4), 1368–1385.

    MathSciNet  Google Scholar 

  • Wu, Q., Jia, C., & Chen, W. (2007). A novel classification-rejection sphere SVMs for multi-class classification problems. In Third international conference on natural computation (ICNC 2007), (vol. 1, pp. 34–38). IEEE.

  • Xu, L., Krzyzak, A., Suen, C. Y. C., Krzyżak, A., & Suen, C. Y. C. (1992). Methods of combining multiple classifiers and their applications to handwriting recognition. IEEE Transactions on Systems, Man, and Cybernetics, 22(3), 418–435.

    Google Scholar 

  • Yuan, M., & Wegkamp, M. H. (2010). Classification methods with reject option based on convex risk minimization. Journal of Machine Learning Research, 11, 111–130.

    MathSciNet  Google Scholar 

  • Zhang, B. (2013). Reliable classification of vehicle types based on cascade classifier ensembles. IEEE Transactions on Intelligent Transportation Systems, 14(1), 322–332.

    Google Scholar 

  • Zhang, C., & Chaudhuri, K. (2014). Beyond disagreement-based agnostic active learning. Advances in Neural Information Processing Systems, 1(January), 442–450.

    Google Scholar 

  • Zhang, C., Wang, W., & Qiao, X. (2018). On reject and refine options in multicategory classification. Journal of the American Statistical Association, 113(522), 730–745.

    MathSciNet  Google Scholar 

  • Zhang, R., & Metaxas, D. (2006). RO-SVM: Support vector machine with reject option for image categorization. In Procedings of the British machine vision conference 2006, (pp. 1–123). British Machine Vision Association.

  • Zheng, E.-H., Zou, C., Sun, J., & Chen, L. (2011). Cost-sensitive SVM with error cost and class-dependent reject cost. International Journal of Computer Theory and Engineering, 3(1), 130–135.

    Google Scholar 

  • Zhou, L., Martínez-Plumed, F., Hernández-Orallo, J., Ferri, C., & Schellaert, W. (2022). Reject before you run: Small assessors anticipate big language models. In CEUR workshop proceedings, (vol. 3169).

  • Zidelmal, Z., Amirou, A., & Belouchrani, A. (2012). Heartbeat classifcation using support vector machines (SVMs) with an embedded reject option. International Journal of Pattern Recognition and Artificial Intelligence, 26(01), 1250001.

    Google Scholar 

  • Ziyin, L., Wang, Z. T., Liang, P. P., Salakhutdinov, R., Morency, L. P., & Ueda, M. (2019). Deep gamblers: Learning to abstain with portfolio theory. Advances in Neural Information Processing Systems, 32(NeurIPS), 1–17.

    Google Scholar 

  • Zong, W., Huang, G. B., & Chen, Y. (2013). Weighted extreme learning machine for imbalance learning. Neurocomputing, 101, 229–242.

    Google Scholar 

  • Zou, C., Zheng, E.-H., Xu, H.-W., & Chen, L. (2011). Cost-sensitive multi-class SVM with reject option: A method for steam turbine generator fault diagnosis. International Journal of Computer Theory and Engineering, 3(1), 77–83.

    Google Scholar 

Download references

Funding

Kilian Hendrickx and Dries Van der Plas received funding from VLAIO (Flemish Innovation & Entrepreneurship) through the Baekeland PhD mandates [HBC.2017.0226] (KH) and [HBC.2019.2615] (DV). Lorenzo Perini received funding from FWO-Vlaanderen, aspirant grant 1166222N. Jesse Davis is partially supported by the KU Leuven research funds [C14/17/070]. Lorenzo Perini, Jesse Davis and Wannes Meert received funding from the Flemish Government under the “Onderzoeksprogramma Artificiële Intelligentie (AI) Vlaanderen” programme.

Author information

Author notes

  1. Kilian Hendrickx and Lorenzo Perini have contributed equally to this work.

Authors and Affiliations

  1. Siemens Digital Industries Software, Leuven, Belgium

    Kilian Hendrickx

  2. Department of Computer Science, KU Leuven, Leuven, Belgium

    Kilian Hendrickx, Lorenzo Perini & Dries Van der Plas

  3. OSG bv, Natus Medical, Kontich, Belgium

    Dries Van der Plas

  4. University of Antwerp, Antwerp, Belgium

    Dries Van der Plas

  5. KU Leuven; Leuven.AI, Leuven, Belgium

    Wannes Meert & Jesse Davis

Authors
  1. Kilian Hendrickx
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lorenzo Perini
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Dries Van der Plas
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Wannes Meert
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jesse Davis
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Concept: JD, WM; Literature Study: KH, LP, DVdP; Categorization: KH, LP, DVdP, WM, JD; Writing - original draft preparation: KH, LP, DVdP; Writing - review and editing: WM, JD; Writing - revision: LP, WM, JD, DVdP, KH; Funding acquisition: WM, JD; Supervision: WM, JD.

Corresponding author

Correspondence to Kilian Hendrickx.

Ethics declarations

Conflict of interest

The authors declare that they have no Conflict of interest.

Ethical approval

Not applicable.

Consent to participate

Not applicable.

Additional information

Editor: Johannes Fürnkranz.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hendrickx, K., Perini, L., Van der Plas, D. et al. Machine learning with a reject option: a survey. Mach Learn 113, 3073–3110 (2024). https://doi.org/10.1007/s10994-024-06534-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10994-024-06534-x

Keywords

Mathematics Subject Classification

Search

Navigation