Text image generation method based on fusion compensation generative adversarial network

Abstract

The invention discloses a text image generation method based on a fusion compensation generative adversarial network, is applied to the field of conditional image generation, and aims to solve the problems that a model is complex, the resolution of a composite image is low and text image feature fusion is not considered in the prior art. According to a fusion compensation generative adversarial network model built by the invention, a sampling block on a generator comprises an affine modulation fusion block, text vectors are introduced into the affine modulation fusion block for multiple times through a condition convolution layer to serve as input, text condition information is repeatedly utilized for multiple times in the feedforward process of the generator and fused into generated image features, compensation of lost information in the feedforward process of the neural network is achieved, and therefore, the model can generate a 256 * 256 resolution image in a single framework at one time, and introduction of an extra network with high calculation cost is avoided.

Description

technical field [0001] The invention belongs to the field of conditional image generation, and in particular relates to an image generation technology for supplementing text conditional information multiple times during the generation process. Background technique [0002] The text generation image task originated in 2016. The content of the task is to convert the natural language description written by humans such as "this bird is black and white, with a short beak" into an image that conforms to the semantics of the text, and its essence is conditional image generation. , that is, image generation with text information as the control, supervision or guidance condition. The main difficulties in generating images from text are: (1) the visual quality of images is low, including clarity, naturalness, and recognition; (2) the semantic similarity between images and given texts is low, that is, the generated images cannot accurately reflect the content of text descriptions. vis...

AI Technical Summary

Problems solved by technology

The main difficulties in generating images from text are: (1) the visual quality of images is low, including clarity, naturalness, and recognition; (2) the semantic similarity between images and given texts is low, that is, the generated images cannot accurately reflect the content of text descriptions. The visual semantic details of

(3) The model is complex, leading to training difficulties, including problems such as unstable training and long training time

[0004] However, compared with the early foundational single-architecture model, the stacking model and the hierarchical nesting model are more complex, and need to rely on additional network structures to improve the semantic richness of the synthetic graph, such as AttnGAN and DM-GAN respectively. Using cross-modal attention mechanism and memory network to introduce word-level fine-grained text vectors to improve semantic fineness, but at the same time further increase the amount of model parameters and calculations

In addition, the above models do not consider the fusion of text and image features, only the simple concatenation of the two is used as the input of the generator, and only one text vector is used. The feedforward process of the generator continuously loses information, resulting in less semantic details of the final composite image.

Images