Text-to-image generation method of adaptive attribute and instance mask embedded graph

Abstract

The invention provides a text-to-image generation method of an adaptive attribute and instance mask embedded graph. The method comprises the following steps: obtaining 64 * 64, 128 * 128 and 256 * 256semantic layouts by using a frame regression network according to an input text; according to the 64 * 64 semantic layout, the sentence embedding vector and the random noise, generating coarse-grained image potential features and a low-resolution 64 * 64 image in a low-resolution generator; forming a pixel-level feature vector in the first high-resolution generator according to the 128 * 128 semantic layout; generating a first fine-grained image potential feature and a high-resolution 128 * 128 image in a first high-resolution generator according to the 128 * 128 semantic layout, the coarse-grained image potential feature and the pixel-level feature vector; and generating a high-resolution 256 * 256 image in a second high-resolution generator according to the 256 * 256 semantic layout, the first fine-grained image potential feature and the pixel-level feature vector. By implementing the method, the resolution of the image is high, the shape constraint of the instance is met, and the attribute characteristics are consistent with the description.

Description

technical field [0001] The invention relates to the technical field of computer vision image generation, in particular to a text-to-image generation method of adaptive attributes and instance mask embedding graphs. Background technique [0002] In recent years, deep learning has achieved good results in the field of text-image generation. Generative confrontation network (GAN), as the most commonly used generation model, jointly learns generators and discriminators; among them, the generator is mainly used to learn pixel distribution and generate realistic images, while the discriminator needs to identify the authenticity of the generated images. Continually confront updates to achieve the final Nash equilibrium. [0003] Generative confrontation network has many input types, such as random noise, semantic segmentation map, sketch map, image pair, scene map, text, etc. Among them, text is the simplest and most operable input form, and more and more researchers It also tend...

AI Technical Summary

Problems solved by technology

[0005] However, the input of existing text-image generation methods is mostly sentence vectors, which lack word-level fine-grained information, and the synthesized images lack instance-level texture features.

In addition, in the process of image generation, the generator easily ignores the spatial interaction relationship between different instances, lacks the instance shape mask constraint, and the synthesized image has problems such as unreasonable instance shape, pixel overlap and occlusion; at the same time, sentence-level The discriminator can only provide coarse-grained training feedback information, and it is difficult to distinguish the visual attributes of word-level instances. As a result, the generative model tends to synthesize the "average" mode of the object instead of the most relevant attribute features, and cannot synthesize real and accurate high-resolution images. rate image

Images