Abstract
It is interesting to edit facial appearance in images to create a desirable facial shape of persons. In this paper, we propose a novel method to modify facial appearance by replacing facial parts between arbitrarily paired images. To this end, our method consists of face segmentation, face reconstruction, mesh deformation and image editing. Given one source and one target image, the target image is first segmented into the front facial region and background image. Secondly, 3D facial models and relevant scene parameters are estimated from both images. Thirdly, the target facial part is replaced with the selected source part on the 3D mesh. Then, the new replaced 3D face is rendered into a facial image. Finally, the new facial image is generated by seamlessly blending the rendered image and background image. The main advantage of this method is that we transfer facial geometric information between images using 3D model, which can deal with arbitrarily paired images with the different facial viewpoint. We present several experimental results to show the effectiveness of our method and comparison with those existing methods to demonstrate that our method is more advantageous and flexible in terms of practical applications.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Chou, J.K., Yang, C.K., Gong, S.D.: Face-off: automatic alteration of facial features. Multimedia Tools Appl. 56(3), 569–596 (2012)
Klum, S., Han, H., Jain, A.K., Klara, B.: Sketch based face recognition: Forensic vs. Composite sketches. In: 2013 International Conference on Biometrics (ICB), pp. 1–8. IEEE, Madrid, Spain (2013)
Google Street View. http://maps.google.com/help/maps/streetview
Blanz, V., Scherbaum, K., Vetter, T., Seidel, H.P.: Exchanging faces in images. Comput. Graph. Forum 23(3), 669–676 (2004)
Bitouk, D., Kumar, N., Dhillon, S., Belhumeur, P., Nayar, S.K.: Face Swap**: automatically replacing faces in photographs. ACM Trans. Graph. (TOG) 27(3), 39:1–39:8 (2008)
Kemelmacher-Shlizerman, I.: Transfiguring portraits. ACM Trans. Graph. (TOG) 35(4), 94:1–94:8 (2016)
Afifi, M., Hussain, K.F., Ibrahim, H.M., Omar, N.M., Video face replacement system using a modified Poisson blending technique. In: 2014 International Symposium on Intelligent Signal Processing and Communication Systems (ISPACS), pp. 205–210. IEEE, Kuching, Malaysia (2014)
Nirkin, Y., Masi, I., Tran, A. T, Hassner, T., Medioni, G.: On Face Segmentation, Face Swap**, and Face Perception. ar**v preprint ar**v:1704.06729, (2017)
Liao, Q., **, X., Zeng, W.: Enhancing the symmetry and proportion of 3D face geometry. IEEE Trans. Visual Comput. Graph. 18(10), 1704–1716 (2012)
Zhao, H., **, X., Huang, X., Chai, M., Zhou, K.: Parametric weight-change resha** for portrait images. IEEE Comput. Graph. Appl. 36 (2016)
Best-Rowden, L., Han, H., Otto, C., Klare, B.F., Jain, A.K.: Unconstrained face recognition: identifying a person of interest from a media collection. IEEE Trans. Inf. Forensics Secur. 9(12), 2144–2157 (2014)
Thies, J., Zollhofer, M., Stamminger, M., Theobalt, C., Niebner, M.: Face2Face: real-time face capture and reenactment of RGB videos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2387–2395. IEEE, Las Vegas, NV, USA (2016)
Cao, C., Weng, Y., Lin, S., Zhou, K.: 3D shape regression for real-time facial animation. ACM Trans. Graph. (TOG) 32(4), 41:1–41:10 (2013)
Li, H., Yu, J., Ye, Y., Bregler, C.: Realtime facial animation with on-the-fly correctives. ACM Trans. Graph. (TOG) 32(4), 42:1–42:10 (2013)
Paysan, P., Knothe, R., Amberg, B.: A 3D face model for pose and illumination invariant face recognition. In: 6th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS 2009), pp. 296–301. IEEE, Genova, Italy (2009)
Jacobson, A., Tosun, E., Sorkine, O.: Mixed finite elements for variational surface modeling. Comput. Graph. Forum 29(5), 1565–1574 (2010)
Wang, H., Cao, J., Liu, X., Wang, J., Fan, T., Hu, J.: Least-squares images for edge-preserving smoothing. Comput. Visual Media 1(1), 27–35 (2015)
Shao, H., Chen, S., Zhao, J., Cui, W., Yu, T.: Face recognition based on subset selection via metric learning on manifold. Front. Inf. Technol. Electron. Eng. 16(12), 1046–1058 (2015)
Oikawa, M.A., Dias, Z., de Rezende Rocha, A., Goldenstein, S.: Manifold learning and spectral clustering for image phylogeny forests. IEEE Trans. Inf. Forensics Secur. 11(1), 5–18 (2016)
Blanz, V., Basso, C., Poggio, T., Vetter, T.: Reanimating faces in images and video. Comput. Graph. Forum 22(3), 641–650 (2003)
Vlasic, D., Brand, M., Pfister, H., Popovic, J.: Face transfer with multilinear models. ACM Trans. Graph. (TOG) 24(3), 426–433 (2005)
Cao, C., Weng, Y., Zhou, S., Tong, Y., Zhou, K.: FaceWarehouse: a 3D facial expression database for visual computing. IEEE Trans. Visual Comput. Graph. 20(3), 413–425 (2014)
Cao, C., Wu, H., Weng, Y., Shao, T., Zhou, K.: Real-time facial animation with image-based dynamic avatars. ACM Trans. Graph. (TOG) 35(4), 1–12 (2016)
Saito, S., Li, T., Li, H.: Real-time facial segmentation and performance capture from RGB input. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9912, pp. 244–261. Springer, Cham (2016). doi:10.1007/978-3-319-46484-8_15
Blanz, V., Vetter, T.: A morphable model for the synthesis of 3D faces. In: Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques, pp. 187–194. ACM Press/Addison-Wesley Publishing Co., New York, USA (1999)
Lin, Y., Wang, S., Lin Q., Tang, F.: Face swap** under large pose variations: a 3D model based approach. In: IEEE International Conference on Multimedia and Expo (ICME), pp. 333–338. IEEE, Melbourne, VIC, Australia (2012)
Song, H., Lv, J., Liu, H., Zhao, Q.: A face replacement system based on 3D face model. In: Deng, Z., Li, H. (eds.) Proceedings of the 2015 Chinese Intelligent Automation Conference. LNEE, vol. 336, pp. 237–246. Springer, Heidelberg (2015). doi:10.1007/978-3-662-46469-4_25
Lin, Y., Lin, Q., Tang, F., Wang, S.: Face replacement with large-pose differences. In: 20th ACM International Conference on Multimedia, pp. 1249–1250. ACM, Nara, Japan (2012)
Tran, A.T., Hassner, T., Masi, I., Medioni, G.: Regressing robust and discriminative 3D morphable models with a very deep neural network. ar**v preprint ar**v:1612.04904 (2017)
Kazemi, V., Sullivan, J.: One millisecond face alignment with an ensemble of regression trees. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1867–1874. IEEE, Columbus, OH, USA (2014)
Huber, P., Hu, G., Tena, R., Mortazavian, P., Koppen W.P., Christmas, W., Ratsch, M., Kittler, J.: A multiresolution 3D Morphable Face Model and fitting framework. In: 11th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, pp. 1–8 (2016)
Sorkine, O.: Least-squares rigid motion using svd. Tech. Notes 120(3), 52 (2009)
Takayama, K., Schmidt, R., Singh, K., Igarashi, T., Boubekeur, T., Sorkine, O.: GeoBrush: interactive mesh geometry cloning. Comput. Graph. Forum 30(2), 613–622 (2011)
Yu, Y., Zhou, K., Xu, D., Shi, X., Bao, H., Guo, B., Shum, H.-Y.: Mesh editing with poisson-based gradient field manipulation. ACM Trans. Graph. (TOG) 23(3), 644–651 (2004)
Schmidt, R., Singh, K.: Drag, drop, and clone: an interactive interface for surface composition. Technical Report CSRG-611, Department of Computer Science, University of Toronto (2010)
Perez, P., Gangnet, M., Blake, A.: Poisson image editing. ACM Trans. Graph. (TOG) 22(3), 313–318 (2003)
Libigl. http://libigl.github.io/libigl/. Accessed 2016
Zhao, J., Tang, M., Tong, R.: Mesh segmentation for parallel decompression on GPU. In: Hu, S.-M., Martin, R.R. (eds.) CVM 2012. LNCS, vol. 7633, pp. 83–90. Springer, Heidelberg (2012). doi:10.1007/978-3-642-34263-9_11
Tang, X., Guo, J., Li, P., Lv, J.: A surgical simulation system for predicting facial soft tissue deformation. Comput. Visual Media 2(2), 163–171 (2016)
Acknowledgements
The research is supported in part by NSFC (61572424) and the People Programme (Marie Curie Actions) of the European Union’s Seventh Framework Programme FP7 (2007–2013) under REA grant agreement No. 612627-“AniNex”. Min Tang is supported in part by NSFC (61572423) and Zhejiang Provincial NSFC (LZ16F020003).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Du, J., Wu, Y., Song, D., Tong, R., Tang, M. (2017). Replacement of Facial Parts in Images. In: Chang, J., Zhang, J., Magnenat Thalmann, N., Hu, SM., Tong, R., Wang, W. (eds) Next Generation Computer Animation Techniques. AniNex 2017. Lecture Notes in Computer Science(), vol 10582. Springer, Cham. https://doi.org/10.1007/978-3-319-69487-0_9
Download citation
DOI: https://doi.org/10.1007/978-3-319-69487-0_9
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-69486-3
Online ISBN: 978-3-319-69487-0
eBook Packages: Computer ScienceComputer Science (R0)