Abstract
The need to modify a photo to reflect the connotations of a text can arise due to multifarious reasons (e.g., a musician might modify a photo in the album cover to better reflect the connotations in her song lyrics). An interesting observation is that different styles of photos convey different feelings. In this paper, we propose the PhotoStylist scheme to effectively modify the style of an input photo to represent the connotations in an input text. Existing methods that aim to transfer emotions into photos rely on an emotion class being provided as input and modify the overall color of photos based on the input emotion class, generating unrealistic colors for many objects in the image. To address these limitations, we design PhotoStylist, a novel deep-learning-based approach, to alter the individual style of each object in the photo in a way that the connotations of the input text are naturally and effectively embedded into the modified photos. Evaluation results on the Amazon Mechanical Turk (MTurk) show that our scheme can achieve output photos significantly closer to the connotations of the input text than the output photos from the state-of-the-art baselines.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Demszky, D., Movshovitz-Attias, D., Ko, J., Cowen, A., Nemade, G., Ravi, S.: GoEmotions: a dataset of fine-grained emotions. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4040–4054 (2020)
Gatys, L.A., Ecker, A.S., Bethge, M.: Image style transfer using convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2414–2423 (2016)
He, L., Qi, H., Zaretzki, R.: Image color transfer to evoke different emotions based on color combinations. SIViP 9(8), 1965–1973 (2015)
Jou, B., Chen, T., Pappas, N., Redi, M., Topkara, M., Chang, S.F.: Visual affect around the world: a large-scale multilingual visual sentiment ontology. In: Proceedings of the 23rd ACM International Conference on Multimedia, pp. 159–168 (2015)
Kant, N., Puri, R., Yakovenko, N., Catanzaro, B.: Practical text classification with large pre-trained language models. ar**v preprint ar**v:1812.01207 (2018)
Li, Y., Liu, M.Y., Li, X., Yang, M.H., Kautz, J.: A closed-form solution to photorealistic image stylization. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 453–468 (2018)
Liu, D., Jiang, Y., Pei, M., Liu, S.: Emotional image color transfer via deep learning. Pattern Recogn. Lett. 110, 16–22 (2018)
Liu, S., Pei, M.: Texture-aware emotional color transfer between images. IEEE Access 6, 31375–31386 (2018)
Luan, F., Paris, S., Shechtman, E., Bala, K.: Deep photo style transfer. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4990–4998 (2017)
Marshall, J., Wang, D.: Mood-sensitive truth discovery for reliable recommendation systems in social sensing. In: Proceedings of the 10th ACM Conference on Recommender Systems, pp. 167–174 (2016)
Marshall, J., Wang, D.: Towards emotional-aware truth discovery in social sensing applications. In: 2016 IEEE International Conference on Smart Computing (SMARTCOMP), pp. 1–8. IEEE (2016)
Miller, G.A.: Wordnet: a lexical database for English. Commun. ACM 38(11), 39–41 (1995)
Reinhard, E., Adhikhmin, M., Gooch, B., Shirley, P.: Color transfer between images. IEEE Comput. Graphics Appl. 21(5), 34–41 (2001)
Seyeditabari, A., Tabari, N., Gholizade, S., Zadrozny, W.: Emotional embeddings: refining word embeddings to capture emotional content of words. ar**v preprint ar**v:1906.00112 (2019)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: 3rd International Conference on Learning Representations, ICLR (2015)
Su, Y.Y., Sun, H.M.: Emotion-based color transfer of images using adjustable color combinations. Soft. Comput. 23(3), 1007–1020 (2019)
Wang, D., Abdelzaher, T., Kaplan, L.: Social Sensing: Building Reliable Systems on Unreliable Data. Morgan Kaufmann, Massachusetts (2015)
Wang, D., Szymanski, B.K., Abdelzaher, T., Ji, H., Kaplan, L.: The age of social sensing. Computer 52(1), 36–45 (2019)
Xu, T., Zhang, P., Huang, Q., Zhang, H., Gan, Z., Huang, X., He, X.: Attngan: fine-grained text to image generation with attentional generative adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1316–1324 (2018)
Yatani, K., Novati, M., Trusty, A., Truong, K.: Analysis of adjective-noun word pair extraction methods for online review summarization. In: Twenty-Second International Joint Conference on Artificial Intelligence, IJCAI (2011)
Yoo, J., Uh, Y., Chun, S., Kang, B., Ha, J.W.: Photorealistic style transfer via wavelet transforms. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 9036–9045 (2019)
Zhang, D.Y., Ni, B., Zhi, Q., Plummer, T., Li, Q., Zheng, H., Zeng, Q., Zhang, Y., Wang, D.: Through the eyes of a poet: Classical poetry recommendation with visual input on social media. In: 2019 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), pp. 333–340. IEEE (2019)
Zhang, D.Y., Shang, L., Geng, B., Lai, S., Li, K., Zhu, H., Amin, M.T., Wang, D.: Fauxbuster: A content-free fauxtography detector using social media comments. In: 2018 IEEE International Conference on Big Data (Big Data), pp. 891–900. IEEE (2018)
Zhang, H., Xu, T., Li, H., Zhang, S., Wang, X., Huang, X., Metaxas, D.N.: Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 5907–5915 (2017)
Zhu, G., Iglesias, C.A.: Computing semantic similarity of concepts in knowledge graphs. IEEE Trans. Knowl. Data Eng. 29(1), 72–85 (2016)
Acknowledgment
This research is supported in part by the National Science Foundation under Grant No. IIS-2008228, CNS-1845639, CNS-1831669, Army Research Office under Grant W911NF-17-1-0409. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Office or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation here on.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Khan, S.K., Zhang, D.(., Kou, Z., Zhang, Y., Wang, D. (2021). PhotoStylist: Altering the Style of Photos Based on the Connotations of Texts. In: Karlapalem, K., et al. Advances in Knowledge Discovery and Data Mining. PAKDD 2021. Lecture Notes in Computer Science(), vol 12712. Springer, Cham. https://doi.org/10.1007/978-3-030-75762-5_51
Download citation
DOI: https://doi.org/10.1007/978-3-030-75762-5_51
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-75761-8
Online ISBN: 978-3-030-75762-5
eBook Packages: Computer ScienceComputer Science (R0)