Abstract
Recently, machine learning has gained enormous momentum and been extensively applied to decision making. However, biases in machine learning models have raised serious concerns. Efforts have been made to reduce biases inherent to training datasets. Unfortunately, data pre-processing aiming to eliminate inherent biases may inadvertently introduce implicit biases that could deflect model training. To address this issue, an in-processing approach, Adversarial Debiasing, has been proposed. Adversarial Debiasing aims to mitigate both dataset biases and implicit biases through its algorithmic process. Nevertheless, model training may not always converge. In this study, we successfully accelerate the convergence of an Adversarial Debiasing model trained on the CelebFaces Attributes dataset using the optimistic Adam optimizer. We also give some discussion on the convergence rate of our proposed framework. We show that given some learning rates, our proposed method adapting the optimistic Adam leads to the convergence of an Adversarial Debiasing model, while one with the Adam optimizer may fail to converge.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data availability
CelebA is a common open dataset available online.
Notes
In addition, OGD has been shown that it can achieve a fast convergence rate to \(\epsilon\)-equilibrium with \(\epsilon = O(\frac{1}{T})\) for the time-averaged parameters; some work studies general normal-form games, in which OGD also achieves a fast convergence rate for each individual player’s regret [12, 13].
In the following experiments, we use a seed 988815313 to initialize the weights of models.
References
Kamiran, F., Calders, T.: Classifying without discriminating. In: 2009 2nd International Conference on Computer, Control and Communication, pp. 1–6 (2009). IEEE
Kamiran, F., Calders, T.: Data preprocessing techniques for classification without discrimination. Knowl. Inf. Syst. 33(1), 1–33 (2012)
Calmon, F., Wei, D., Vinzamuri, B., Natesan Ramamurthy, K., Varshney, K.R.: Optimized pre-processing for discrimination prevention. Adv. Neural Inf. Process. Syst. 30 (2017)
Kamishima, T., Akaho, S., Sakuma, J.: Fairness-aware learning through regularization approach. In: 2011 IEEE 11th International Conference on Data Mining Workshops, pp. 643–650 (2011). IEEE
Zhang, B.H., Lemoine, B., Mitchell, M.: Mitigating unwanted biases with adversarial learning. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pp. 335–340 (2018)
Daskalakis, C., Ilyas, A., Syrgkanis, V., Zeng, H.: Training gans with optimism (2017). ar**v preprint ar**v:1711.00141
Hardt, M., Price, E., Srebro, N.: Equality of opportunity in supervised learning. Adv. Neural Inf. Process. Syst. 29 (2016)
McCarthy, M. B., Narayanan, S.: Fairness–accuracy tradeoff: activation function choice in a neural network. AI Ethics (2023)
Neumann, J.V.: Zur theorie der gesellschaftsspiele. Math. Ann. 100(1), 295–320 (1928)
Kodali, N., Abernethy, J., Hays, J., Kira, Z.: On convergence and stability of gans (2017). ar**v preprint ar**v:1705.07215
Mertikopoulos, P., Papadimitriou, C., Piliouras, G.: Cycles in adversarial regularized learning. In: Proceedings of the Twenty-Ninth Annual ACM-SIAM Symposium on Discrete Algorithms, pp. 2703–2717 (2018). SIAM
Syrgkanis, V., Agarwal, A., Luo, H., Schapire, R.E.: Fast convergence of regularized learning in games. Adv. Neural Inf. Process. Syst. 28 (2015)
Rakhlin, A., Sridharan, K.: Online learning with predictable sequences. In: Conference on Learning Theory, pp. 993–1019 (2013). PMLR
Pedreshi, D., Ruggieri, S., Turini, F.: Discrimination-aware data mining. In: Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 560–568 (2008)
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., Zemel, R.: Fairness through awareness. In: Proceedings of the 3rd Innovations in Theoretical Computer Science Conference, pp. 214–226 (2012)
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. Adv. Neural Inf. Process. Syst. 27 (2014)
Shalev-Shwartz, S., et al.: Online learning and online convex optimization. Found. Trends® Mach. Learn. 4(2), 107–194 (2012)
Freund, Y., Schapire, R.E.: Adaptive game playing using multiplicative weights. Games Econ. Behav. 29(1–2), 79–103 (1999)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: Proceedings of the 3rd International Conference on Learning Representations (2015)
Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein generative adversarial networks. In: International Conference on Machine Learning, pp. 214–223 (2017). PMLR
Anagnostides, I., Panageas, I., Farina, G., Sandholm, T.: On last-iterate convergence beyond zero-sum games (2022). ar**v preprint ar**v:2203.12056
Liu, Z., Luo, P., Wang, X., Tang, X.: Deep learning face attributes in the wild. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3730–3738 (2015)
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
On behalf of all the authors, the corresponding author states that there is no conflict of interest.
Additional information
Part of the work was done while the first author was a master student at NYCU.
Appendix: Adam
Appendix: Adam
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Cheng, YC., Chen, PA., Chen, FC. et al. Adversarial learning with optimism for bias reduction in machine learning. AI Ethics (2023). https://doi.org/10.1007/s43681-023-00356-8
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s43681-023-00356-8