Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Fig. 1.
figure 1

Overview of our pose transfer pipeline: given an input image and a target pose we use DensePose [1] to drive the generation process. This is achieved through the complementary streams of (a) a data-driven predictive module, and (b) a surface-based module that warps the texture to UV-coordinates, interpolates on the surface, and warps back to the target image. A blending module combines the complementary merits of these two streams in a single end-to-end trainable framework.

Full size image

1 Introduction

Deep models have recently shown remarkable success in tasks such as face [2], human [3,4,5], or scene generation [6, 7], collectively known as “neural synthesis”. This opens countless possibilities in computer graphics applications, including cinematography, gaming and virtual reality settings. At the same time, the potential malevolent use of this technology raises new research problems, including the detection of forged images or videos [1. The war** module uses DensePose surface correspondence and inpainting to generate a new view of the person, while the predictive module is a generic, black-box, generative model conditioned on the DensePose outputs for the input and the target.

These modules corresponding to two parallel streams have complementary merits: the predictive module successfully exploits the dense conditioning output to generate plausible images for familiar poses, delivering superior results to those obtained from sparse, landmark-based conditioning; at the same time, it cannot generalize to new poses, or transfer texture details. By contrast, the war** module can preserve high-quality details and textures, allows us to perform inpainting in a uniform, canonical coordinate system, and generalizes for free for a broad variety of body movements. However, its body-, rather than clothing-centered construction does not take into account hair, hanging clothes, and accessories. The best of both worlds is obtained by feeding the outputs of these two blocks into a blending module trained to fuse and refine their predictions using a combination of reconstruction, adversarial, and perceptual losses in an end-to-end trainable framework.

The DensePose module is common to both streams and delivers dense correspondences between an image and a surface-based model of the human body. It does so by firstly assigning every pixel to one of 24 predetermined surface parts, and then regressing the part-specific surface coordinates of every pixel. The results of this system are encoded in three output channels, comprising the part label and part-specific UV surface coordinates. This system is trained discriminatively and provides a simple, feed-forward module for dense correspondence from an image to the human body surface. We omit further details, since we rely on the system of [1] with minor implementation differences described in Sect. 4.

Having outlined the overall architecture of our system, in Sects. 3.1 and 3.3 we present in some more detail our components, and then turn in Sect. 3.4 to the loss functions used in their training. A thorough description of architecture details is left to the supplemental material. We start by presenting the architecture of the predictive stream, and then turn to the surface-based stream, corresponding to the upper and lower rows of Fig. 1, respectively.

3.1 Predictive Stream

The predictive module is a conditional generative model that exploits the DensePose system results for pose transfer. Existing conditional models indicate the target pose in the form of heat-maps from keypoint detectors [4], or part segmentations [3]. Here we condition on the concatenation of the input image and DensePose results for the input and target images, resulting in an input of dimension \(256\, \times \, 256 \times 9\). This provides conditioning that is both global (part-classification), and point-level (continuous coordinates), allowing the remaining network to exploit a richer source of information.

The remaining architecture includes an encoder followed by a stack of residual blocks and a decoder at the end, along the lines of [28]. In more detail, this network comprises (a) a cascade of three convolutional layers that encode the \(256 \times 256 \times 9\) input into \(64 \times 64 \times 256\) activations, (b) a set of six residual blocks with \(3 \times 3 \times 256 \times 256 \) kernels, (c) a cascade of two deconvolutional and one convolutional layer that deliver an output of the same spatial resolution as the input. All intermediate convolutional layers have \(3{\times }3\) filters and are followed by instance normalization [36] and ReLU activation. The last layer has tanh non-linearity and no normalization.

3.2 War** Stream

Our war** module performs pose transfer by performing explicit texture map** between the input and the target image on the common surface UV-system. The core of this component is a Spatial Transformer Network (STN) [37] that warps according to DensePose the image observations to the UV-coordinate system of each surface part; we use a grid with \(256\ {\times }\ 256 \) UV points for each of the 24 surface parts, and perform scattered interpolation to handle the continuous values of the regressed UV coordinates. The inverse map** from UV to the output image space is performed by a second STN with a bilinear kernel. As shown in Fig. 3, a direct implementation of this module would often deliver poor results: the part of the surface that is visible on the source image is typically small, and can often be entirely non-overlap** with the part of the body that is visible on the target image. This is only exacerbated by DensePose failures or systematic errors around the part seams. These problems motivate the use of an inpainting network within the war** module, as detailed below.

Fig. 2.
figure 2

Supervision signals for pose transfer on the war** stream: the input image on the left is warped to intrinsic surface coordinates through a spatial transformer network driven by DensePose. From this input, the inpainting autoencoder has to predict the appearance of the same person from different viewpoints, when also warped to intrinsic coordinates. The loss functions on the right penalize the reconstruction only on the observed parts of the texture map. This form of multi-view supervision acts like a surrogate for the (unavailable) appearance of the person on the full body surface.

Full size image

Inpainting Autoencoder. This model allows us to extrapolate the body appearance from the surface nodes populated by the STN to the remainder of the surface. Our setup requires a different approach to the one of other deep inpainting methods [33], because we never observe the full surface texture during training. We handle the partially-observed nature of our training signal by using a reconstruction loss that only penalizes the observed part of the UV map, and lets the network freely guess the remaining domain of the signal. In particular, we use a masked \(\ell _1\) loss on the difference between the autoencoder predictions and the target signals, where the masks indicate the visibility of the target signal.

We observed that by its own this does not urge the network to inpaint successfully; results substantially improve when we accompany every input with multiple supervision signals, as shown in Fig. 2, corresponding to UV-wrapped shots of the same person at different poses. This fills up a larger portion of the UV-space and forces the inpainting network to predict over the whole texture domain. As shown in Fig. 3, the inpainting process allows us to obtain a uniformly observed surface, which captures the appearance of skin and tight clothes, but does not account for hair, skirts, or apparel, since these are not accommodated by DensePose’s surface model.

Our inpainting network is comprised of N autoencoders, corresponding to the decomposition of the body surface into N parts used in the original DensePose system [1]. This is based on the observation that appearance properties are non-stationary over the body surface. Propagation of the context-based information from visible to invisible parts that are entirely occluded at the present view is achieved through a fusion mechanism that operates at the level of latent representations delivered by the individual encoders, and injects global pose context in the individual encoding through a concatenation operation.

In particular, we denote by \(\mathcal {E}_i\) the individual encoding delivered by the encoder for the i-th part. The fusion layer concatenates these obtained encodings into a single vector which is then down-projected to a 256-dimensional global pose embedding through a linear layer. We pass the resulting embedding through a cascade of ReLU and Instance-Norm units and transform it again into an embedding denoted by \(\mathcal {G}\).

Then the i-th part decoder receives as an input the concatenation of \(\mathcal {G}\) with \(\mathcal {E}_i\), which combines information particular to part i, and global context, delivered by \(\mathcal {G}\). This is processed by a stack of deconvolution operations, which delivers in the end the prediction for the texture of part i.

3.3 Blending Module

The blending module’s objective is to combine the complementary strengths of the two streams to deliver a single fused result, that will be ‘polished’ as measured by the losses used for training. As such it no longer involves an encoder or decoder unit, but rather only contains two convolutional and three residual blocks that aim at combining the predictions and refining their results.

In our framework, both predictive and war** modules are first pretrained separately and then finetuned jointly during blending. The final refined output is obtained by learning a residual term added to the output of the predictive stream. The blending module takes an input consisting of the outputs of the predictive and the war** modules combined with the target dense pose.

Fig. 3.
figure 3

War** module results. For each sample, the top row shows interpolated textures obtained from DensePose predictions and projected on the surface of the 3D body model. The bottom row shows the same textures after inpainting in the UV space.

Full size image

3.4 Loss Functions

As shown in Fig. 1, the training set for our network comes in the form of pairs of input and target images, \(\varvec{x}\), \(\varvec{y}\) respectively, both of which are of the same person-clothing, but in different poses. Denoting by \(\varvec{\hat{y}}=G(\varvec{x})\) the network’s prediction, the difference between \(\varvec{\hat{y}},\varvec{y}\) can be measured through a multitude of loss terms, that penalize different forms of deviation. We present them below for completeness and ablate their impact in practice in Sect. 4.

Reconstruction Loss. To penalize reconstruction errors we use the common \(\ell _1\) distance between the two signals: \(\Vert \varvec{\hat{y}} -\varvec{y}\Vert _1\). On its own, this delivers blurry results but is important for retaining the overall intensity levels.

Perceptual Loss. As in Chen and Koltun [6], we use a VGG19 network pretrained for classification [38] as a feature extractor for both \(\varvec{\hat{y}},\varvec{y}\) and penalize the \(\ell _2\) distance of the respective intermediate feature activations \(\varPhi ^{v}\) at 5 different network layers \(v=1,\ldots ,N\):

$$\begin{aligned} \mathcal {L}_{\mathrm {p}}(\varvec{y},\varvec{\hat{y}}) = \sum _{v=1}^{N}\Vert {\varPhi }^{v}(\varvec{y}) - {\varPhi }^{v}(\varvec{\hat{y}})\Vert _2. \end{aligned}$$
(1)

This loss penalizes differences in low- mid- and high-level feature statistics, captured by the respective network filters.

Style Loss. As in [28], we use the Gram matrix criterion of [25] as an objective for training a feedforward network. We first compute the Gram matrix of neuron activations delivered by the VGG network \(\varPhi \) at layer v for an image \(\varvec{x}\):

$$\begin{aligned} \mathcal {G}^{v}(\varvec{x})_{c,c'} = \sum _{h,w}\varPhi ^{v}_{c}(\varvec{x})[h,w]\varPhi ^{v}_{c'}(\varvec{x})[h,w] \end{aligned}$$
(2)

where h and w are horizontal and vertical pixel coordinates and c and \(c'\) are feature maps of layer v. The style loss is given by the sum of Frobenius norms for the difference between the per-layer Gram matrices \(\mathcal {G}^{v}\) of the two inputs:

$$\begin{aligned} \mathcal {L}_{\mathrm {style}}(\varvec{y},\varvec{\hat{y}}) = \sum _{v=1}^B \Vert \mathcal {G}^{v}(\varvec{y}) - \mathcal {G}^{v}(\varvec{\hat{y}})\Vert _{F}. \end{aligned}$$
(3)

Adversarial Loss. We use adversarial training to penalize any detectable differences between the generated and real samples. Since global structural properties are largely settled thanks to DensePose conditioning, we opt for the patchGAN [12] discriminator, which operates locally and picks up differences between texture patterns. The discriminator [7, 12] takes as an input \(\varvec{z}\), a combination of the source image and the DensePose results on the target image, and either the target image \(\varvec{y}\) (real) or the generated output (fake) \(\varvec{\hat{y}}\). We want fake samples to be indistinguishable from real ones – as such we optimize the following objective:

$$\begin{aligned} L_{\text {GAN}} = \underbrace{\frac{1}{2}\mathbb {E}_{\varvec{z}}\left[ l(D(\varvec{z},\varvec{y})-1)\right] + \frac{1}{2}\mathbb {E}_{\varvec{z}}\left[ l(D(\varvec{z},\varvec{\hat{y}}))\right] }_{\mathrm {Discriminator}} + \underbrace{\frac{1}{2}\mathbb {E}_{\varvec{z}}\left[ l(D(G(\varvec{z})-1))\right] }_{\mathrm {Generator}}, \end{aligned}$$
(4)

where we use \(l(x) = x^2\) as in the Least Squares GAN (LSGAN) work of [4.1 Implementation Details

DensePose Estimator. We use a fully convolutional network (FCN) similar to the one used as a teacher network in [1]. The FCN is a ResNet-101 trained on cropped person instances from the COCO-DensePose dataset. The output consists of 2D fields representing body segments (I) and \(\{U,V\}\) coordinates in coordinate spaces aligned with each of the semantic parts of the 3D model.

Training Parameters. We train the network and its submodules with Adam optimizer with initial learning rate \(2\,{\cdot }\,10^{-4}\) and \(\beta _1\,{=}\,0.5\), \(\beta _2\,{=}\,0.999\) (no weight decay). For speed, we pretrain the predictive module and the inpainting module separately and then train the blending network while finetuning the whole combined architecture end-to-end; DensePose network parameters remain fixed. In all experiments, the batch size is set to 8 and training proceeds for 40 epochs. The balancing weights \(\lambda \) between different losses in the blending step (described in Sect. 3.4) are set empirically to \(\lambda _{\ell _1}=1\), \(\lambda _{\text {p}}=0.5\), \(\lambda _{\text {style}}=5\,{\cdot }\,10^{5}\), \(\lambda _{\text {GAN}}=0.1\).

Table 1. Quantitative comparison with the state-of-the-art methods on the DeepFashion dataset [19] according to the Structural Similarity (SSIM) [40], Inception Score (IS) [41] and detection score (DS) [5] metrics. Our best structure model corresponds to the \(\ell _1\) loss, the highest realism model corresponds to the style loss training (see the text and Table 4). Our balanced model is trained using the full combination of losses.
Full size table

4.2 Evaluation Metrics

As of today, there exists no general criterion allowing for adequate evaluation of the generated image quality from the perspective of both structural fidelity and photorealism. We therefore adopt a number of separate structural and perceptual metrics widely used in the community and report our joint performance on them.

Structure. The geometry of the generations is evaluated using the perception-correlated Structural Similarity metric (SSIM) [40]. We also exploit its multi-scale variant MS-SSIM [43] to estimate the geometry of our predictions at a number of levels, from body structure to fine clothing textures.

Image Realism. Following previous works, we provide the values of Inception scores (IS) [41]. However, as repeatedly noted in the literature, this metric is of limited relevance to the problem of within-class object generation, and we do not wish to draw strong conclusions from it. We have empirically observed its instability and high variance with respect to the perceived quality of generations and structural similarity. We also note that the ground truth images from the DeepFashion dataset have an average IS of 3.9, which indicates low degree of realism of this data according to the IS metric (for comparison, IS of CIFAR-10 is 11.2 [41] with best image generation methods achieving IS of 8.8 [2]).

In addition, we perform additional evaluation using detection scores (DS) [5] reflecting the similarity of the generations to the person class. Detection scores correspond to the maximum of confidence of the PASCAL-trained SSD detector [44] in the person class taken over all detected bounding boxes.

Fig. 4.
figure 4

Qualitative comparison with the state-of-the-art Deformable GAN (DSC) method of [5]. Each group shows the input, the target image, DSC predictions [5], predictions obtained with our full model. We observe that even though our cloth texture is occasionally not as sharp, we better retain face, gender, and skin color information.

Full size image

4.3 Comparison with the State-of-the-Art

We first compare the performance of our framework to a number of recent methods proposed for the task of keypoint guided image generation or multi-view synthesis. Table 1 shows a significant advantage of our pipeline in terms of structural fidelity of obtained predictions. This holds for the whole range of tested network configurations and training setups (see Table 4). In terms of perceptional quality expressed through IS, the output generations of our models are of higher quality or directly comparable with the existing works. Some qualitative results of our method (corresponding to the balanced model in Table 1) and the best performing state-of-the-art approach [5] are shown in Fig. 4.

We have also performed a user study on Amazon Mechanical Turk, following the protocol of [5]: we show 55 real and 55 generated images in a random order to 30 users for one second. As the experiment of [5] was done with the help of fellow researchers and not AMT users, we perform an additional evaluation of images generated by [5] for consistency, using the official public implementation. We perform three evaluations, shown in Table 1: Realism asks users if the image is real or fake. Anatomy asks if a real, or generated image is anatomically plausible. Pose shows a pair of a target and a generated image and asks if they are in the same pose. The results (correctness, in %) indicate that generations of [5] have higher degree of perceived realism, but our generations show improved pose fidelity and higher probability of overall anatomical plausibility.

4.4 Effectiveness of Different Body Representations

In order to clearly measure the effectiveness of the DensePose-based conditioning, we first compare to the performance of the ‘black box’, predictive module when used in combination with more traditional body representations, such as background/foreground masks, body part segmentation maps or body landmarks.

Fig. 5.
figure 5

Typical failures of keypoint-based pose transfer (top) in comparison with DensePose conditioning (bottom) indicate disappearance of limbs, discontinuities, collapse of 3D geometry of the body into a single plane and confusion in ordering in depth.

Full size image
Table 2. On effectiveness of different body representations as a ground for pose transfer. The DensePose representation results in the highest structural quality.
Full size table

As a segmentation map we take the index component of DensePose and use it to form a one-hot encoding of each pixel into a set of class specific binary masks. Accordingly, as a background/foreground mask, we simply take all pixels with positive DensePose segmentation indices. Finally, following [5] we use [39] to obtain body keypoints and one-hot encode them.

In each case, we train the predictive module by concatenating the source image with a corresponding representation of the source and the target poses which results in 4 input planes for the mask, 27 for segmentation maps and 21 for the keypoints.

The corresponding results shown in Table 2 demonstrate a clear advantage of fine-grained dense conditioning over the sparse, keypoint-based, or coarse, segmentation-based, representations.

Complementing these quantitative results, typical failure cases of keypoint-based frameworks are demonstrated in Fig. 5. We observe that these shortcomings are largely fixed by switching to the DensePose-based conditioning.

4.5 Ablation Study on Architectural Choices

Table 3 shows the contribution of each of the predictive module, war** module, and inpainting autoencoding blocks in the final model performance. For these experiments, we use only the reconstruction loss \(\mathcal {L}_{\ell _1}\), factoring out fluctuations in the performance due to instabilities of GAN training. As expected, including the war** branch in the generation pipeline results in better performance, which is further improved by including the inpainting in the UV space. Qualitatively, exploiting the inpainted representation has two advantages over the direct war** of the partially observed texture from the source pose to the target pose: first, it serves as an additional prior for the fusion pipeline, and, second, it also prevents the blending network from generating clearly visible sharp artifacts that otherwise appear on the boarders of partially observed segments of textures.

Table 3. Contribution of each of the functional blocks of the framework
Full size table

4.6 Ablation Study on Supervision Objectives

In Table 4 we analyze the role of each of the considered terms in the composite loss function used at the final stage of the training, while providing indicative results in Fig. 6.

Fig. 6.
figure 6

Effects of training with different loss terms and their weighted combinations.

Full size image
Table 4. Comparison of different loss terms used at the final stage of the training. The perceptual loss is best correlated with the structure, and the style loss with IS. The combined model provides an optimal balance between the extreme solutions.
Full size table

The perceptual loss \(\mathcal {L}_p\) is most correlated with the image structure and least correlated with the perceived realism, probably due to introduced textural artifacts. At the same time, the style loss \(\mathcal {L}_{\text {style}}\) produces sharp and correctly textured patterns while hallucinating edges over uniform regions. Finally, adversarial training with the loss \(\mathcal {L}_{\text {GAN}}\) tends to prioritize visual plausibility but often disregarding structural information in the input. This justifies the use of all these complimentary supervision criteria in conjunction, as indicated in the last entry of Table 4.

5 Conclusion

In this work we have introduced a two-stream architecture for pose transfer that exploits the power of dense human pose estimation. We have shown that dense pose estimation is a clearly superior conditioning signal for data-driven human pose estimation, and also facilitates the formulation of the pose transfer problem in its natural, body-surface parameterization through inpainting. In future work we intend to further pursue the potential of this method for photorealistic image synthesis [2, 6] as well as the treatment of more categories.