Abstract
We present a new method which generalizes subspace learning based on eigenvalue and generalized eigenvalue problems. This method, Roweis Discriminant Analysis (RDA) named after Sam Roweis, is a family of infinite number of algorithms where Principal Component Analysis (PCA), Supervised PCA (SPCA), and Fisher Discriminant Analysis (FDA) are special cases. One of the extreme special cases, named Double Supervised Discriminant Analysis (DSDA), uses the labels twice and it is novel. We propose a dual for RDA for some special cases. We also propose kernel RDA, generalizing kernel PCA, kernel SPCA, and kernel FDA, using both dual RDA and representation theory. Our theoretical analysis explains previously known facts such as why SPCA can use regression but FDA cannot, why PCA and SPCA have duals but FDA does not, why kernel PCA and kernel SPCA use kernel trick but kernel FDA does not, and why PCA is the best linear method for reconstruction. Roweisfaces and kernel Roweisfaces are also proposed generalizing eigenfaces, Fisherfaces, supervised eigenfaces, and their kernel variants. We also report experiments showing the effectiveness of RDA and kernel RDA on some benchmark datasets.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Ghojogh, B., et al.: Feature selection and feature extraction in pattern analysis: a literature review. ar**v preprint ar**v:1905.02845 (2019)
Ghojogh, B., Crowley, M.: Unsupervised and supervised principal component analysis: Tutorial. ar**v preprint ar**v:1906.03148 (2019)
Ghojogh, B., Karray, F., Crowley, M.: Fisher and kernel Fisher discriminant analysis: Tutorial. ar**v preprint ar**v:1906.09436 (2019)
Schölkopf, B., Smola, A., Müller, K.-R.: Kernel principal component analysis. In: Gerstner, W., Germond, A., Hasler, M., Nicoud, J.-D. (eds.) ICANN 1997. LNCS, vol. 1327, pp. 583–588. Springer, Heidelberg (1997). https://doi.org/10.1007/BFb0020217
Schölkopf, B., Smola, A., Müller, K.R.: Nonlinear component analysis as a kernel eigenvalue problem. Neural Comput. 10(5), 1299–1319 (1998)
Hofmann, T., Schölkopf, B., Smola, A.J.: Kernel methods in machine learning. Ann. Stat. 1171–1220 (2008)
Mika, S., Rätsch, G., Weston, J., Schölkopf, B., Müller, K.R.: Fisher discriminant analysis with kernels. In: Proceedings of the 1999 IEEE Signal Processing Society Workshop on Neural Networks for Signal Processing IX, pp. 41–48. IEEE (1999)
Barshan, E., Ghodsi, A., Azimifar, Z., Jahromi, M.Z.: Supervised principal component analysis: visualization, classification and regression on subspaces and submanifolds. Pattern Recogn. 44(7), 1357–1371 (2011)
Gretton, A., Bousquet, O., Smola, A., Schölkopf, B.: Measuring statistical dependence with Hilbert-Schmidt norms. In: Jain, S., Simon, H.U., Tomita, E. (eds.) ALT 2005. LNCS (LNAI), vol. 3734, pp. 63–77. Springer, Heidelberg (2005). https://doi.org/10.1007/11564089_7
Turk, M.A., Pentland, A.P.: Face recognition using eigenfaces. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 1991, pp. 586–591. IEEE (1991)
Yang, M.H., Ahuja, N., Kriegman, D.: Face recognition using kernel eigenfaces. In: Proceedings of 2000 International Conference on Image Processing, vol. 1, pp. 37–40. IEEE (2000)
Belhumeur, P.N., Hespanha, J.P., Kriegman, D.J.: Eigenfaces vs. Fisherfaces: recognition using class specific linear projection. IEEE Trans. Pattern Anal. Mach. Intell. 19(7), 711–720 (1997)
Yang, M.H.: Kernel eigenfaces vs. kernel Fisherfaces: face recognition using kernel methods. In: Proceedings of the Fifth IEEE International Conference on Automatic Face and Gesture Recognition, pp. 215–220 (2002)
Wang, J.: Geometric Structure of High-Dimensional Data and Dimensionality Reduction. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-27497-8
Ghojogh, B., Karray, F., Crowley, M.: Eigenvalue and generalized eigenvalue problems: tutorial. ar**v preprint ar**v:1903.11240 (2019)
Parlett, B.N.: The symmetric eigenvalue problem. Classics Appl. Math. 20 (1998)
Ye, J.: Least squares linear discriminant analysis. In: Proceedings of the 24th International Conference on Machine Learning, pp. 1087–1093. ACM (2007)
Zhang, Z., Dai, G., Xu, C., Jordan, M.I.: Regularized discriminant analysis, ridge regression and beyond. J. Mach. Learn. Res. 11(Aug), 2199–2228 (2010)
Alperin, J.L.: Local Representation Theory: Modular Representations as an Introduction to the Local Representation Theory of Finite Groups, vol. 11. Cambridge University Press, Cambridge (1993)
Ghojogh, B., Karray, F., Crowley, M.: Roweis discriminant analysis: a generalized subspace learning method. ar**v preprint ar**v:1910.05437 (2019)
Mika, S., Rätsch, G., Weston, J., Schölkopf, B., Smola, A.J., Müller, K.R.: Invariant feature extraction and classification in kernel spaces. In: Advances in Neural Information Processing Systems, pp. 526–532 (2000)
Turk, M., Pentland, A.: Eigenfaces for recognition. J. Cogn. Neurosci. 3(1), 71–86 (1991)
Stanković, R.S., Falkowski, B.J.: The Haar wavelet transform: its status and achievements. Comput. Electr. Eng. 29(1), 25–44 (2003)
Wang, Y.Q.: An analysis of the Viola-Jones face detection algorithm. Image Process. On Line 4, 128–148 (2014)
Li, B., Zha, H., Chiaromonte, F.: Contour regression: a general approach to dimension reduction. Ann. Stat. 33(4), 1580–1616 (2005)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Ghojogh, B., Karray, F., Crowley, M. (2020). Generalized Subspace Learning by Roweis Discriminant Analysis. In: Campilho, A., Karray, F., Wang, Z. (eds) Image Analysis and Recognition. ICIAR 2020. Lecture Notes in Computer Science(), vol 12131. Springer, Cham. https://doi.org/10.1007/978-3-030-50347-5_29
Download citation
DOI: https://doi.org/10.1007/978-3-030-50347-5_29
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-50346-8
Online ISBN: 978-3-030-50347-5
eBook Packages: Computer ScienceComputer Science (R0)