Abstract
Cross-media learning is a new hot topic in multimedia content analysis and retrieval. Because multimedia data of different modalities are heterogeneous in feature space and there exists the well-know semantic gap, one of the most challenging issues for cross-media learning is to mine underlying semantics and estimate cross-media correlation. In this paper we propose a cross-media semantics mining approach based on Sparse Canonical Correlation Analysis and relevance feedback. First, we analyze sparse canonical correlation between low-level feature matrices of different modalities in training stage, and construct a Multimodal Sparse Subspace where both canonical correlation and most meaningful features are preserved; then based on geometric distance in the subspace we estimate cross-media correlation and enable cross-media retrieval; also we provide long-term relevance feedback strategy for performance optimization. Our approach is tested with general multimedia data, including image, audio and text. Experiment and comparison results are encouraging and show that the performance of our approach is effective.
This work is supported by National Natural Science Foundation of China (No.61003127).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Yang, Y., Zhuang, Y., Wu, F., Pan, Y.: Harmonizing Hierarchical Manifolds for Multimedia Document Semantics Understanding and Cross-media Retrieval. IEEE Transactions on Multimedia 10(3), 437–446 (2008)
Zhang, H., Liu, X.: Boosting Multimodal Semantic Understanding by Local Similarity Adaptation and Global Correlation Propagation. In: Qiu, G., Lam, K.M., Kiya, H., Xue, X.-Y., Kuo, C.-C.J., Lew, M.S. (eds.) PCM 2010, Part I. LNCS, vol. 6297, pp. 148–158. Springer, Heidelberg (2010)
Zhuang, Y., Yang, Y., Wu, F.: Mining Semantic Correlation of Heterogeneous Multimedia Data for Cross-media Retrieval. IEEE Transactions on Multimedia 10(2), 221–229 (2008)
Lew, M.: Content Based Multimedia Information Retrieval: State of the Art and Challenges. ACM Transactions on Multimedia Computing, Communications and Applications 2(1), 1–19 (2006)
He, X., Ma, W.Y., Zhang, H.J.: Learning an Image Manifold for Retrieval. In: Proceedings of ACM Multimedia Conference (2004)
Zhang, H., Weng, J.: Measuring Multi-modality Similarities Via Subspace Learning for Cross-Media Retrieval. In: Zhuang, Y.-T., Yang, S.-Q., Rui, Y., He, Q. (eds.) PCM 2006. LNCS, vol. 4261, pp. 979–988. Springer, Heidelberg (2006)
Hotelling, H.: Relations Between Two Sets of Variates. Biometrika, 321–377 (1936)
Witten, D.M., Tibshirani, R.: Extensions of sparse canonical correlation analysis, with applications to genomic data. Statistical Applications in Genetics and Molecular Biology 8(1) (2009)
Torres, D.A.: Using sparse CCA for vocabulary selection. M.S. University of California, San Diego (2009)
Torres, D.A., Turnbull, D., Barrington, L., Sriperumbudur, B.K., Lanckriet, G.: Finding Musically Meaningful Words by Sparse CCA. In: NIPS Workshop on Music, Brain & Cognition (2007)
Zhang, R., Zhang, Z.: Effective Image Retrieval based on Hidden Concept Discovery in Image Database. IEEE Transactions on Image Processing 16(2), 562–572 (2007)
Yang, Y., Nie, F., Xu, D., Luo, J., Zhuang, Y., Pan, Y.: A multimedia retrieval framework based on semi-supervised ranking and relevance feedback. IEEE Transactions on Pattern Analysis and Machine Intelligence 34(4), 723–742 (2012)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Zhang, H., Liu, X. (2012). Cross-Media Semantics Mining Based on Sparse Canonical Correlation Analysis and Relevance Feedback. In: Lin, W., et al. Advances in Multimedia Information Processing – PCM 2012. PCM 2012. Lecture Notes in Computer Science, vol 7674. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-34778-8_71
Download citation
DOI: https://doi.org/10.1007/978-3-642-34778-8_71
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-34777-1
Online ISBN: 978-3-642-34778-8
eBook Packages: Computer ScienceComputer Science (R0)