Skip to content

index.md

Latest commit

 

History

History
62 lines (44 loc) · 3.57 KB

index.md

File metadata and controls

62 lines (44 loc) · 3.57 KB
Raw
layout default

Abstract

Pose-guided person image generation is to transform a source person image to a target pose. This task requires spatial manipulations of source data. However, Convolutional Neural Networks are limited by lacking the ability to spatially transform the inputs. In this paper, we propose a differentiable global-flow local-attention framework to reassemble the inputs at the feature level. Specifically, our model first calculates the global correlations between sources and targets to predict flow fields. Then, the flowed local patch pairs are extracted from the feature maps to calculate the local attention coefficients. Finally, we warp the source features using a content-aware sampling method with the obtained local attention coefficients. The results of both subjective and objective experiments demonstrate the superiority of our model. Besides, additional results in video animation and view synthesis show that our model is applicable to other tasks requiring spatial transformation.

Network Architecture

Our network spatially transforms the image features using a Global-Flow Local-Attention framework. First, the Flow Field Estimator is used to obtain the global flow fields which indicate the approximate sampling positions. Octocat

Then, the local attention operation is performed for each local patch in the target features centered at position l. This operation allows the network sampling vivid textures from the source features according to the target pose. Octocat

Results and Applications

Pose-based Person Image Generation

Form Left to Right: Source, Target Pose, Target Image, DefGAN, VU-Net, Pose-Attn, Intr-Flow, Ours.

Image Animations

  • DeepFashion: In this task, we use a source image and a sequence of target poses to generate the result video. This task can be seen as an extension of pose-based person image generation, which yields continuous videos.

From Left to Right: Real Video, Extracted Pose, Animation Results.

  • Face Animation: Given a source face and a sequence of edge images, our model generates the result video with specific motions.

Left: Input Source Image and Edge Sequence, Right: Animation results.

View Synthesis

  • ShapeNet: View synthesis requires generating novel views of objects or scenes based on arbitrary input views. In this task, we generate multi-images with different view points based on a single input source image.

Form Left to Right: Source, Results of Appearance flow, Ours, and Ground-truth images.